Samsung R&D Institute China-Beijing (SRC-B) clinched first place in two different tracks of the 18th International Workshop on Semantic Evaluation 2024 (SemEval-2024) competition, succeeding in the subtasks for “Task 3: The Competition of Multimodal Emotion Cause Analysis in Conversations” and “Task 8: Multidomain, Multimodel, and Multilingual Machine-Generated Text Detection.” Furthermore, both teams’ system papers were accepted for presentation at the associated conference.

Task 3: The Competition of Multimodal Emotion Cause Analysis in Conversations

Understanding emotions is crucial for human-like artificial intelligence, holding significant application values. In customer service scenarios, large language models (LLMs) with emotion comprehension capabilities can help companies effectively analyze customer feedback and sentiments. Emotion comprehension enhances various areas, such as personalized learning and education. It also helps in generating tailored content that emotionally resonates with the audience in chatting, content creation, and marketing, aside from understanding and responding to emotions that play a crucial part in mental health support services. Emotion Cause Analysis, which identifies the underlying reasons for emotions, is vital for comprehending human behavior.

The “Subtask 1: Textual Emotion-Cause Pair Extraction in Conversations” involves extracting emotion-cause pairs from the given conversation solely based on text, where the emotion cause is defined and annotated as a textual span. Meanwhile, Subtask 2 Multimodal Emotion-Cause Pair Extraction in Conversations expands the definition of Subtask 1, considering three modalities (text, video, and audio), extracting all emotion-cause pairs in the conversation at the utterance level. The proposed multistage framework generates emotion and extracts the emotion causal pairs with the generated emotion. The dataset is created from the popular series “Friends,” containing numerous instances of almost-real daily life scenarios and multiparty conversation.

In the first stage, LLaMA2 was utilized, modifying InstructERC to tune the model on the task data with Low-Rank Adaptation (LoRA) parameter efficient tuning. After extracting the emotion category of each utterance in a conversation, competitive emotion recognition performance was achieved in comparison with the existing published works. The predicted emotion functions as guidance for cause span extraction in Subtask 1 and cause utterance extraction in Subtask 2. To extract a causal span for Subtask 1, the SRC-B team utilized the state-of-the-art traditional regular-sized model Multi-Task learning framework extracting Emotions and emotion Cause (MuTEC) and employed a two-stream attention model to extract the emotion causal pairs given the target emotion for Subtask 2.

By combining traditional LLMs’ impressive general abilities and the dedicated structural/learning target design of the specific domain model to achieve high performance, the SRC-B teams’ approach achieved first place for both two subtasks in the competition.

SRC-B Team in SemEval-2024 Task 3

Task 8: Multidomain, Multimodel, and Multilingual Machine-Generated Text Detection

LLMs are increasingly prevalent and readily accessible, leading to a surge in machine-generated content across various platforms, such as news, social media, education, and academia. However, concerns have arisen regarding their potential misuse, including the spread of misinformation and disruptions in education. Consequently, there is a pressing need to develop automated systems capable of identifying machine-generated text to mitigate these risks.

The objective of Task 8 is to encourage participants to utilize models to improve the detection of machine-generated texts, offering three subtasks across two text generation paradigms: (1) full text, where the text is entirely written by a human or generated by a machine, and (2) mixed text, where a machine-generated text is edited by a human or a human-written text is paraphrased by a machine.

SRC-B teams secured two first-place standings in two subtasks of Task 8. The first is Subtask B, which focuses on Multi-Way Machine-Generated Text Classification, aiming to discern whether a text originates from humans or specific LLMs. The other one is Subtask C, which targets Human-Machine Mixed Text Detection, aiming to delineate the boundary between human-written and machine-generated sections within a mixed text.

SRC-B teams maximized the potential of LLMs for detecting machine-generated texts and conducted comprehensive experiments with LLM-based models. They achieved state-of-the-art performance in both subtasks, establishing a benchmark for future studies in this promising area. In addition, they explored factors influencing LLMs’ text detection capabilities, including additional layers on top of LLMs, segment loss function design, and pretraining techniques. Their findings aim to offer valuable insights for future research in this domain.

SRC-B Team in SemEval-2024 Task 8