Computer-aided design (CAD) is the most popular design method for most physical products today. Engineers use CAD to transform 2D sketches into 3D models, which they can then test and refine before sending the final version to the production line. However, the software is extremely complex to learn and has thousands of commands to choose from. It takes a huge amount of time and practice to become truly proficient with the software.
MIT engineers want to make CAD easier to learn with an AI model that uses CAD software just like a human would. Given a 2D sketch of an object, the model quickly creates a 3D version by clicking buttons and file options, much like an engineer would use software.
The MIT team has created a new dataset called VideoCAD that contains over 41,000 examples of creating 3D models in CAD software. By learning from these videos, which illustrate step-by-step how various shapes and objects are constructed, the new AI system can now operate CAD software just like a human.
With VideoCAD, the team is working on an AI-enabled “CAD second pilot.” They envision that such a tool could not only create 3D versions of a design, but also work with the user to suggest next steps or automatically perform build sequences that would otherwise be tedious and time-consuming to click manually.
“Artificial intelligence has the potential to increase engineer productivity and also make CAD software more accessible to more people,” says Ghadi Nehme, a graduate student in MIT's Department of Mechanical Engineering.
“This is important because it lowers the barrier to entry into the design industry, helping people without years of CAD training more easily create 3D models and use their creativity,” adds Faez Ahmed, associate professor of mechanical engineering at MIT.
Ahmed and Nehme, along with graduate student Brandon Man and postdoc Ferdous Alam, will present their work at the Neural Information Processing Systems (NeurIPS) Conference in December.
Click after click
The team's new work expands on the latest developments in AI-powered UI agents – tools trained to use programs to perform tasks such as automatically collecting information online and organizing it in an Excel spreadsheet. Ahmed's group wondered whether such UI agents could be designed to use CAD software, which includes many more features and requires much more complex tasks than the average UI agent can handle.
In their new work, the team aimed to design an AI-powered UI agent that takes the reins of a CAD program and creates a 3D version of a 2D sketch, click by click. To do this, the team first looked through an existing dataset of human CAD-designed objects. Each object in the dataset contains a sequence of high-level design commands such as “sketch line”, “circle”, and “extrude” that were used to build the final object.
However, the team realized that high-level commands alone were not enough to train the AI agent to actually use the CAD software. A true agent must also understand the details of each action. For example: Which sketch area should I choose? When should you enlarge? And how much of the sketch should he draw? To fill this gap, researchers developed a system that translates high-level commands into user interface interactions.
“For example, suppose we drew a sketch by drawing a line from point 1 to point 2,” says Nehme. “We've translated these high-level actions into UI actions, which means we say go from here in pixels, click, and then go to the second place in pixels and click, with the 'line' action selected.”
Ultimately, the team generated over 41,000 videos of human-designed CAD objects, each of which was annotated in real time for specific clicks, mouse drags, and other keyboard actions that were originally performed by a human. They then fed all this data into a model they developed to understand the connections between UI actions and CAD object generation.
Once trained on this dataset, which is called VideoCAD, the new AI model can take a 2D sketch as input and directly drive the CAD software by clicking, dragging and selecting tools to construct a full 3D shape. The objects varied in complexity, from simple cantilevers to more complex house designs. The team is training the model on more complex shapes and anticipates that both the model and the dataset will one day enable collaborative CAD pilots for designers across disciplines.
“VideoCAD is a valuable first step toward AI assistants that help onboard new users and automate repetitive modeling work following known patterns,” says Mehdi Ataei, who was not involved in the study but is a senior researcher at Autodesk Research, which develops new design software tools. “This is just the beginning, and I would be excited to see cross-CAD successors that offer richer operations such as assemblies and mates, and a more realistic and unstructured human workflow.”
