This blog will showcase my progress and feedback from my mentor throughout my graduation semester at Secura.
The graduation semester spans 18 weeks, from September 2, 2024, to January 24, 2025.
Weeks 1 and 2 (September 2, 2024 - September 13, 2024)
- Project Planning: Created a comprehensive project plan in line with the university's requirements.
- Functional Document: Began drafting a functional document aimed at explaining the project to non-technical stakeholders.
- University Supervisor Meeting: Held my first meeting with the university supervisor to discuss the project's direction and goals.
- LLM Fine-tuning: Started fine-tuning a local LLM model to handle LaTeX content translation.
- Interview: I interviewed my company mentor asking him some questions regarding my project at Secura.
- Research: I finished the first 2 research question. First research question helped me to choose an open-source LLM which is Llama 3.1 for my project, and the second research question analyzed the hardware requirements for that mode which is dual 4090 GPU.
Feedback from Joel:
- The use cases and functional requirements are good, but there are some important sections missing.
- The functional and technical design should explain the current setup and what it will look like when the project is finished.
- Add a section called "Justification of Choice for Research Environment." This should explain why you chose the operating system, focusing on things like how future-proof and user-friendly it is.
- Include a section about hardware. Since this is the functional design, explain what hardware is needed and for what purpose, but there’s no need to include technical details.
- Add a section on security. This should cover things like who can access the system, the security policies in place, and how physical access is managed.
- It would be helpful to use a MoSCoW chart to clearly define the project goals and priorities.
Feedback from Gayatri:
- We discussed the overall project and its scope, and Gayatri emphasized the need to focus more on the infrastructure side rather than the software.
- The initial draft of the project plan included 10 research questions, but Gayatri recommended reducing the number to a maximum of 5 to keep the project manageable.
- Gayatri suggested shifting the project's focus more towards the infrastructure aspect, ensuring it aligns with the company's needs, and university requirements.
Weeks 3 and 4 (September 16, 2024 - September 27, 2024)
- Project Plan & Functional Design Updates: Added additional details to both the project plan and the functional design document.
- Backend Server Research: Conducted research on backend server options and concluded that it would be valuable to include in my in-depth research.
- MoSCoW Board Discussion: Had a meeting with Joel to discuss the MoSCoW board and prioritize project elements.
- First Company Visit: Gayatri visited me at the Secura Amsterdam office. We discussed my progress with my company mentor.
- Functional Document: Describe things in a non-technical way.
Feedback from Joel:
- Invest some time to look at the EU AI Act and how it is used in the project.
Feedback from Gayatri:
- Gayatri liked the structure of the first version of the project plan and mentioned that it is almost perfect.
- To complete the project plan, Gayatri recommended adding a network diagram and a list of professional products for each learning outcome.
- Create design challenge document.
- Create logbook.
Weeks 5 and 6 (September 30, 2024 - October 11, 2024)
- Functional Document: I finished the functional document that describes things in a non-technical way, and it was approved by my company mentor.
- Technical Design Document: I created the first draft of technical design document.
- Research: I finished the 3rd research question that talks about vector databases, and I concluded that I will be using Chroma DB for my project.
- Design Challenge Document: I finished the design challenge document.
- EU AI Act Document: I created a document discussing the EU AI Act and how it applies in my project.
Feedback from Joel:
- He approved the functional design document and advised to start translating it into technical design document.
- The first draft of technical design document is missing the details of how I plan to build the LLM server. I still do not know the details. Therefore, I suggested that I finish my research first, then create the technical document, Joel agreed on this approach.
Weeks 7 and 8 (October 14, 2024 - November 01, 2024)
- Research: I finished the 4th research question which talked about the backend and frontend frameworks for my project. I ended up choosing vLLM for the backend, and Open web UI for the frontend.
- Portfolio: I started creating the Portfolio by combining every professional product in one place. I submitted the first version of the portfolio.
- Reading: I invested some time reading about new trends in open source AI models. This can help me create the advise report later this semester.
- Cloud Prices: Joel asked me to research the prices of GPUs using the cloud and choose the best one for my project in case there is another delay in the hardware. Therefore, I created a document detailing the price of many possible services.
Feedback from Joel:
- You have answered sub-questions 1 to 4 so far, but the elaboration is rather short, and it would be good to read more about the "how" and "why" behind your choices, as I believe you have done the analysis to determine which option is best.
- For example, "Chroma DB’s user-friendly interface and many plugins make it easier to set up and use, saving time during development," but the "how" and "why" are missing, so more details about the kind of plugins and their specific benefits would be helpful; another example is "Open Web UI was chosen for its user-friendly design," which would benefit from an explanation of why this design is better compared to other options.
- Mohammed is always offering to help, but it should not limit him in his eagerness to analyze, test or execute things without asking.
Feedback from Gayatri:
- You are doing well in showcasing all learning outcomes with proper evidence.
- Try to add some theory in the Project Approach section, preferably in the form of diagrams or other visual elements.
- For each sub-question: Explain its DOT framework and strategy, Remove the conclusion for each question, Add a solution for each question instead.
- For Conclusions and recommendations: Include them as separate topics for the overall report, Do not add them for each question individually.
- The rest is progressing well.
Weeks 9 and 10 (November 4, 2024 - November 15, 2024)
- Literature Review: I read about LLM optimization techniques. This can help me improve the speed of the LLM inference.
- Cybersecurity Event in Utrecht: My company mentor Joel advised me to visit a cybersecurity event in Utrecht. It was a special experiences meeting professionals in the field of cybersecurity and AI.
- Meeting with IT department: Me and Joel had a meeting with the IT specialist discussing the status of the LLM hardware and how I will be able to access it.
- Mid-term reviews: I asked Joel to fill in the mid-term review for Fontys. I also filled in one myself reflecting on my progress so far.
Feedback from Joel:
- Find out how Secura writes test plan document and create one for your project.
- Joel said that Ralph approved paying for the cloud as a temporary solution. Therefore, start building you findings using a cloud provider.
Weeks 11 and 12 (November 18, 2024 - November 29, 2024)
- Mid-term Return Day Pitch: I presented my progress to my 2 Fontys assessors and to some students. I was also able to learn from other graduates who presented their work.
- Assessors Feedback: I worked on the feedback received from the 2 assessors.
- Rent a GPU: I used the cloud to rent a GPU and build my product on it as a temporary solution.
- Demo: I showed Joel a demo for the LLM deployed on the cloud.
Feedback from Mehrzad:
- Show the approach better (DOT framework usage in details)
- Show agenda for the presentation
- Show page numbers
- Include a page for sprints activities
- Explain why your solution is better than using ready products like ChatGPT.
Weeks 13 and 14 (December 02, 2024 - December 13, 2024)
- LLM Design Plan: I created an LLM Design Plan that explains to the IT department what is the LLM machine expected to do and what operating system should it run. I also explained what type of connections does it need and when should it be disconnected completely from internet access. The document also explains who will be the administrators for the application.
- Technical Implementation Guide: I created a Technical Implementation Guide document that explains in details all the steps needed to host the frontend and the backend in an Ubuntu 20.04 server. The steps are the following:
- Download CUDA Toolkit
- Install cuDNN
- Install Python 3.11
- Create a Virtual Environment
- Install & Run the Backend (vLLM)
- Install & Run the Frontend (Open Web UI)
- Access the Application
- Access the Server: I got SSH connection to the LLM machine. I installed the backend and the frontend and connected them with each other. I had issues installing Nvidia CUDA toolkit and cuDNN. However, I eventually successfully installed everything needed. Every issue and its fix is documented very well.
- Secura Christmas Party: I got the chance to attend the company's Christmas party, it was nice meeting and connecting with other Secura employees.
Feedback from Joel:
- Joel helped me to improve the documents.
- Joel encountered error when trying to upload big document. I fixed this issue by increasing the LLM's capacity to handle around 10 thousand words per request.
Weeks 15 and 16 (December 16, 2024 - January 10, 2025) + Christmas Break
- Meeting with Joel Regarding the Deliverables: I had a meeting with my mentor to discuss the deliverables for both company and university.
- I canceled the cloud subscription: Now that we have a powerful server with dual 4090 locally, I canceled the cloud subscription.
- Discussed the LLM's Accuracy with Joel: I had a meeting with Joel to discuss the current accuracy of the LLM and discuss plans on how to improve it. Joel asked me to find new ways to improve the accuracy.
- New Embedding Model: To improve the accuracy, I installed a better and more powerful embedding model. One of the drawbacks is it would need more time to answer and it would consume more of the GPUs. However, Joel said that these issues can be over seen if the accuracy is getting better.
Feedback from Joel:
- Create Back up plans.
Weeks 17 and 18 (January 13, 2025 - January 24, 2025) + Christmas Break
- Meeting with Stakeholders: I had multiple meetings with Joel, Ralph, and Paul to discuss the future of this project. I showed them that the RAG and translator are working.
- Fine Tuning the LLM: Paul asked me if I can fine tune the LLM on their own data. I used many Secura reports to create a dataset for the fine tuning. I managed to create a fine tuned model on only 1300 examples. I explained to them that in order to have decent quality, they need more examples to include.
- Documenting: Joel, Ralph, and Paul mentioned that I should document every detail to make it easier for whoever comes after me working on the project.
