Majeed Kazemitabaar

Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners' perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools.

We developed CodeAid, an LLM-powered programming assistant designed to offer students timely and personalized feedback without directly revealing code solutions. This tool aids in answering conceptual questions, generating pseudo-code, and suggesting corrections for incorrect code. We deployed CodeAid in a large class of 700 students and conducted a thematic analysis of its 8,000 usages, supplemented by weekly surveys and student interviews. Further feedback was obtained from eight programming educators. Results showed that most students used CodeAid for understanding concepts and debugging, though some students directly asked for code solutions. While educators valued its educational merits, they also highlighted concerns about occasional inaccuracies and the potential for students to rely on tools like ChatGPT.

We then concluded with four key design considerations for AI assistants in educational contexts, centered around four main stages of a student's help-seeking process:

• Exploiting Unique Advantages of AI: Decision to use the AI tool, emphasizing the unique advantages of AI over other resources.
• Designing the AI Querying Interface: Query formulation, providing context, and balancing user-friendliness with meta-cognitive engagement.
• Balancing the Directness of AI Responses: Nature of AI responses, managing directness, scaffolding type, and learning engagement.
• Supporting Trust, Transparency and Control: Post-response actions, ensuring accuracy, trust, transparency, and control.

We studied the impact of Large Language Model (LLM)-based code generators, such as OpenAI Codex, on novice programmers (ages 10-17). In a controlled experiment involving 69 novices working on 45 Python tasks, we found that using Codex led to a 1.15x increase in code-authoring completion rate and 1.8x higher scores, without diminishing manual code-modification capabilities. Interestingly, those with prior Codex exposure had slightly improved performance in evaluations a week later. A deeper dive into data from 33 participants who used Codex revealed various ways they interacted with the tool. We identified four coding strategies: AI Single Prompt, AI Step-by-Step, Hybrid, and Manual coding. The AI Single Prompt strategy yielded the highest correctness in code-authoring but struggled in code-modification tasks. Our findings highlighted both the potentials and pitfalls of LLMs in educational settings, emphasizing the need for balanced integration and curriculum development.

We designed CodeStruct, an intermediary programming environment aimed at assisting learners transition from block-based programming, like Scratch, to text-based languages such as Python. CodeStruct bridges the learning curve between these two paradigms, offering design features that significantly reduce completion times and help requests when compared to a direct transition. In a study with 26 high school students, results indicated that those using CodeStruct had a smoother transition with a decrease in data-type and syntax issues, especially when initially aided by a structured editor. However, once they moved to an unstructured editor, the rate of syntax errors increased, though they still outperformed peers who transitioned directly.

A key challenge in developing and debugging custom embedded systems is understanding their behavior, particularly at the boundary between hardware and software. Bifröst automatically instruments and captures the progress of the user's code, variable values, and the electrical and bus activity occurring at the interface between the processor and the circuit it operates in. This data is displayed in a linked visualization that allows navigation through time and program execution, enabling comparisons between variables in code and signals in circuits.

GestureBlocks incorporates a demonstrate-edit-review Machine Learning pipeline for authoring sensor-based gestures into Microsoft MakeCode and allows novices to program behaviors using both data-driven and conventional paradigms.

When programmers face new frameworks they usually rely on example codes to learn about the API and accomplish their tasks. This work investigates and analyzes the activities performed by programmers when such sample codes are being used for task completion.