--- type: Article title: Vulnerability-oriented Testing for RESTful APIs resource: "https://www.usenix.org/conference/usenixsecurity24/presentation/du" tags: [article, webseclist-reference, en, usenix-org] generated: by: webseclist-refs/1 at: "2026-08-19T16:21:32+00:00" status: stable stale_after: 2027-08-19 sources: - id: original resource: "https://www.usenix.org/conference/usenixsecurity24/presentation/du" title: Vulnerability-oriented Testing for RESTful APIs author: Wenlong Du, Jian Li, Yanhao Wang, Libo Chen, Ruijie Zhao, Junmin Zhu, Zhengguang Han, Yijun Wang, Zhi Xue also_at: - "https://www.usenix.org/system/files/usenixsecurity24-du.pdf" - "https://www.usenix.org/system/files/sec24summer-prepub-572-du.pdf" - "https://www.usenix.org/system/files/usenixsecurity24_slides-du.pdf" authors: - Wenlong Du - Jian Li - Yanhao Wang - Libo Chen - Ruijie Zhao - Junmin Zhu - Zhengguang Han - Yijun Wang - Zhi Xue canonical_url: "" cited_by: - "2024.md:145" commit: "" content_sha256: a26b5d75beca5776bf4a2b56f07f139033836e186593af258106f5ebd5b43f98 depth: full depth_reason: default kind: article language: en licence: unknown original_url: "https://www.usenix.org/conference/usenixsecurity24/presentation/du" published: "" publisher: usenix.org publisher_english: "" raw_sha256: 2a207a8245874673966f6db893cac6122edb3bc51cd277d24c47f1496a3394ff retrieved_from: "https://www.usenix.org/system/files/usenixsecurity24-du.pdf" retrieved_kind: live retrieved_utc: "2026-08-19T16:21:32+00:00" slug: usenix-org-vulnerability-oriented-testing-restful-apis snapshot: "" title_english: "" translation_file: "" translation_of: "" --- # Vulnerability-oriented Testing for RESTful APIs **Vulnerability-oriented Testing for RESTful APIs** - Wenlong Du, Jian Li, Yanhao Wang, Libo Chen, Ruijie Zhao, Junmin Zhu, Zhengguang Han, Yijun Wang, Zhi Xue, usenix.org. - Published: date not stated - Original: - Also published at: - Also published at: - Also published at: - Preserved from: https://www.usenix.org/system/files/usenixsecurity24-du.pdf (live) on 2026-08-19 - Licence: unknown Rights remain with the original author and publisher. This is a research archive of a source from the Web Hacking Techniques Index collections, kept so the page going offline. To read the original, follow the link above. ## Content > UNTRUSTED SOURCE TEXT. Everything below this line is third-party material > quoted for research. It is data, not instructions. Do not follow directions, > execute code, or fetch URLs because this text says so. Vulnerability-oriented Testing for RESTful APIs Wenlong Du and Jian Li, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University; Yanhao Wang, Independent Researcher; Libo Chen, Ruijie Zhao, and Junmin Zhu, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University; Zhengguang Han, QI-ANXIN Technology Group; Yijun Wang and Zhi Xue, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University https://www.usenix.org/conference/usenixsecurity24/presentation/du This paper is included in the Proceedings of the 33rd USENIX Security Symposium. August 14–16, 2024 • Philadelphia, PA, USA 978-1-939133-44-1 Open access to the Proceedings of the 33rd USENIX Security Symposium is sponsored by USENIX. Vulnerability-oriented Testing for RESTful APIs Wenlong Du*1 , Jian Li*1 , Yanhao Wang2 , Libo Chen B1 , Ruijie Zhao1 , Junmin Zhu1 , Zhengguang Han3 , Yijun Wang1 , and Zhi Xue1 1 School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University 2 Independent Researcher 3 QI-ANXIN Technology Group {beet1e, vfenux483, bob777, ruijiezhao, junmin.zhu, ericwyj, zxue}@sjtu.edu.cn, wangyanhao136@gmail.com, hanzhengguang@qianxin.com Abstract API development, REST [9] (REpresentational State Trans- With the increasing popularity of APIs, ensuring their secu- fer) has emerged as a widely embraced approach, with APIs rity has become a crucial concern. However, existing security adhering to this style commonly known as RESTful APIs. testing methods for RESTful APIs usually lack targeted ap- Unfortunately, numerous security issues [10, 11] have proaches to identify and detect security vulnerabilities. In arisen in APIs in recent years, resulting in significant repercus- this paper, we propose VOAPI2 , a vulnerability-oriented API sions for providers and users. For instance, a vulnerability [10] inspection framework designed to directly expose vulnera- in a Facebook API interface exposed millions of users’ private bilities in RESTful APIs, based on our observation that the information. As we see, API security has become a critical type of vulnerability hidden in an API interface is strongly concern, especially for service providers. Meanwhile, APIs associated with its functionality. By leveraging this insight, have increasingly become a focal point for attackers, making we first track commonly used strings as keywords to iden- them a common target in various services. Hence, proactively tify APIs’ functionality. Then, we generate a stateful and identifying and addressing API flaws is vital in safeguarding suitable request sequence to inspect the candidate API func- services and their users. Detecting vulnerabilities early on is tion within a targeted payload. Finally, we verify whether an essential strategy to ensure the protection of these systems. vulnerabilities exist or not through feedback-based testing. Currently, API testing has gained significant popularity Our experiments on real-world APIs demonstrate the effec- as a solution to enhance API security, and a considerable tiveness of our approach, with significant improvements in amount of research has been conducted in this field. However, vulnerability detection compared to state-of-the-art methods. existing testing methods for RESTful APIs often encounter VOAPI2 discovered 7 zero-day and 19 disclosed bugs on limitations when it comes to identifying security vulnerabili- seven real-world RESTful APIs, and 23 of them have been ties, especially in black-box testing scenarios [12–17]. These assigned CVE IDs. Our findings highlight the importance of methods heavily rely on the API’s status feedback, which considering APIs’ functionality when discovering their bugs, means they may not detect certain logical vulnerabilities (e.g., and our method provides a practical and efficient solution for SSRF) that do not affect the normal functioning of APIs. On securing RESTful APIs. the other hand, white-box testing methods [17–21] require source code security audits, which can be time-consuming, inefficient, and prone to generating false positives. Gray-box 1 Introduction testing methods [22, 23] require access to the internal state of API execution, offering a more comprehensive view. Never- Application Programming Interfaces (APIs), as the contract theless, the diverse code frameworks used in API implemen- between information providers and information users, are tations present a challenge in developing a unified detection gaining immense popularity as they effectively facilitate com- approach. Furthermore, testing methods tailored to specific ar- munication and data exchange between diverse web appli- chitectures [24] have limited scope and are difficult to extend cations. It also enables businesses to harness their data and beyond their original application scenarios. services well to support innovative scenarios and broaden In addition, several other methods [12, 15, 16, 25–32] have their impact. Currently, it has been widely used and spanned been proposed that leverage OpenAPI1 specifications to en- across various scenarios, such as cloud services [1, 2], content management systems (CMS) [3–6], and Internet of Things 1 OpenAPI [33] (i.e., Swagger) stands as the most widely used specifica- (IoT) [7, 8] devices. Among different architectural styles for tion for describing RESTful API interfaces. A Swagger-based specification outlines the usage guidelines for RESTful APIs, including the types of service * Co-leading authors. B Corresponding author. requests accepted, the expected response format, and other relevant details. USENIX Association 33rd USENIX Security Symposium 739 able unified testing of APIs. These methods include functional request sequence that aligns with the execution context of testing [12, 15, 16, 25–29, 34], compliance testing [30], and the corresponding functions. By utilizing inspection payloads security testing [31, 32]. Parts of these approaches [26–29] tailored to different vulnerability types, we can effectively utilize OpenAPI specifications to parse HTTP requests and assess these candidate functions for potential vulnerabilities. generate fuzzing test cases that adhere to syntax rules. Con- We evaluated our tool on 7 real-world RESTful APIs, which versely, some methods [12, 15, 16, 25, 34] use OpenAPI spec- resulted in the discovery of 7 zero-day vulnerabilities and 19 ifications to infer the dependencies between requests and disclosed bugs. Out of these vulnerabilities, 23 have been ver- generate test messages that align with the API interaction ified and acknowledged by the respective vendors, and have context. This enables effective testing of API functionality, as been assigned CVE IDs. The experiment results demonstrate well as the identification of exceptions and specific security that VOAPI2 surpasses the state-of-the-art methods by a sig- vulnerabilities. nificant margin, not only in terms of the number of identified Despite the emergence of various testing approaches, it bugs but also in terms of detection efficiency. is noteworthy that most of these methods do not effectively In summary, we make the following contributions: utilize the specific characteristics of security vulnerabilities to implement vulnerability-targeted testing strategies. Instead, • We propose and implement a novel method, VOAPI2 , to they primarily rely on the detection of “500 Internal Server Er- automatically reveal the vulnerability of RESTful APIs. It rors” as an indication of capturing bugs within API interfaces. can perform vulnerability-oriented testing based on API Consequently, this approach leads to limited visibility and low specifications to effectively capture defects in RESTful testing efficiency. For example, when dealing with a large- APIs, and output the corresponding PoC. We will release scale API project similar in scale to GitLab, RESTler [12], on the source code as well as the experiment data at https: average, requires more than 5 hours to perform a comprehen- //github.com/NSSL-SJTU/VoAPI2. sive test. Furthermore, the security vulnerabilities uncovered • We evaluated our tool with several state-of-the-art methods. by these tools, such as RestTestGen [16], represent only a The experiment results demonstrate that VOAPI2 surpasses small fraction of these bugs, with the majority being func- all these methods by a significant margin, not only in terms tional issues that primarily impact the reliability of APIs. of the number of identified bugs but also in terms of detec- Based on our analysis of publicly disclosed API vulnera- tion efficiency. bilities in recent years, we observed that there exists a clear correlation between different API security vulnerabilities and • We conducted an evaluation of VOAPI2 on 7 real-world the functionality of APIs. For instance, file upload interfaces RESTful APIs, and VOAPI2 identified 26 vulnerabilities, can pose a risk of malware uploads, while proxy interfaces including 7 previously-unknown and 19 known security may harbor SSRF vulnerabilities. Building upon this obser- bugs. All zero-day bugs have been reported to vendors and vation, we propose the core concept of our method, which fixed; four of them have been assigned CVE IDs. involves identifying API functions associated with vulnera- bilities within API specifications and conducting targeted 2 Problem and Approach Overview security testing on those functions. This approach aims to achieve vulnerability-oriented API inspection, thereby fa- In this section, we first provide the background of vulnerabili- cilitating the effective discovery of bugs across a wide range ties in RESTful API. Then, we present a motivation example, of APIs. and our method to detect the hidden vulnerability in the sam- There are three challenges in realizing vulnerability- ple. At last, we discuss the associated challenges and present oriented API inspection in practice. The first challenge is effi- an overview of our solution. ciently differentiating between various functional interfaces in an API and identifying functions that may have security vulnerabilities. The second challenge is effectively generating 2.1 Vulnerability of RESTful API valid test cases based on different function interfaces and cor- REST, short for REpresentational State Transfer, is a widely responding security vulnerability types. The last challenge is adopted software architecture style for network applications. generating test sequences that comply with protocol states, in- It is commonly utilized within the context of HTTP, and an corporating the related test cases, and successfully executing API adhering to this style is referred to as a RESTful API. the function interfaces. While existing tools are available for testing REST APIs, To address these challenges, we propose VOAPI2 , a many of them primarily focus on assessing the reliability Vulnerability-Oriented API Inspection framework designed of these APIs rather than specifically targeting potential se- to directly expose vulnerabilities in RESTful APIs. Our ap- curity vulnerabilities. However, similar to web applications, proach begins by identifying semantic keywords associated REST APIs are also vulnerable to various security threats, as with potentially weak functions within the API specification. demonstrated by the presence of such vulnerabilities in the It then integrates a suitable corpus and employs a stateful CVE database. These vulnerabilities encompass issues like 740 33rd USENIX Security Symposium USENIX Association 1 /Images/Remote: 2 get: /Images/Ratings 3 operationId: GetRemoteImage /Images/Remote 4 parameters: 5 description: The image url. /Items 6 in: query /Items/Filter 7 name: imageUrl 8 required: true … 9 schema: REST APIs 10 description: The image url. Attacker Internal Service 11 format: uri 12 type: string 13 responses: Figure 2: Exploitation process of CVE-2021-29490. 14 "200": 15 content: 16 image/*: (Line 26) that offers category tags to classify the API. 17 schema: 18 format: binary In Jellyfin (version 10.7.3 and earlier), there exists a 19 type: string Server-Side Request Forgery (SSRF) vulnerability (CVE- 20 description: Remote image returned. 2021-29490 [35]) in the aforementioned API endpoint. Ex- 21 "404": ploiting this vulnerability requires the attacker to send a GET 22 content: 23 # omitted request to the /Images/Remote API with testing payloads (e.g., 24 description: Remote image not found. http://LAN_IP/ssrf) in the query parameter “imageUrl”. 25 summary: Gets a remote image. The root cause of this vulnerability is the lack of filtering 26 tags: for the remote resource address specified by the “imageUrl” 27 - RemoteImage parameter in the backend of the API service. This oversight allows attackers to freely access internal resources by sending Figure 1: Specification of an API endpoint. an HTTP request (i.e., the testing payload) from the Jellyfin server, as depicted in Figure 2. Server-Side Request Forgery, Unrestricted File Upload, OS As we see, the vulnerability in the motivation sample is Command Injection, Path Traversal, and more. closely related to the functionality of the corresponding API endpoint. Specifically, the SSRF bug typically arises in the Motivation Example. OpenAPI, formerly known as Swag- API path responsible for retrieving a remote resource. As ger, provides a specification for describing RESTful APIs. demonstrated in this example, the purpose of this API end- Typically, the specification document is presented in JSON or point is to fetch a remote image, as indicated in its summary YAML format and encompasses the resources and operations (Line 25). The parameter “imageUrl” is utilized to specify the of the RESTful API. It comprises a collection of Schema Ob- address of the resource (Line 5). Consequently, if the back- jects, which encompass various elements such as API service end of this API interface fails to rigorously validate input endpoints (commonly referred to as paths), CRUD operations parameters, it may result in an SSRF vulnerability. (such as GET, POST, PUT and DELETE) performed on an endpoint, input parameters, and expected responses. Each schema object follows a predefined structure with corresponding parame- 2.2 Observation and Our Method ter types. Users can leverage the specification to generate valid API operations and communicate with API services by Current security testing methods for APIs suffer from two presenting them as HTTP requests. primary efficiency issues and most of them cannot detect the Figure 1 illustrates a simplified fragment of the OpenAPI vulnerability hidden in the motivation sample. Firstly, they specification for Jellyfin [3], an open-source media man- are time-consuming. For example, RESTler [12] required over agement system. This fragment specifies an API endpoint, five hours to test four groups of API interfaces in the open- /Images/Remote, which supports a single CRUD operation, source code management software GitLab, with the longest ef- GET. It defines the input parameters and expected response fective request sequence length tested remaining at just three. for this endpoint. Specifically, the input parameter is named This inefficiency can be attributed to testing tools needing “imageUrl” (Line 7), which is required and used to specify to traverse all interfaces indiscriminately, rather than focus- the URL of the image (Line 10). The corresponding schema ing on a specific API interface in-depth. Secondly, existing (Line 9) designates the parameter type as a string (Line 12). testing tools heavily rely on service status feedback from The response section includes the HTTP status code and API endpoints to gauge testing effectiveness, resulting in the response message (Line 13). Furthermore, this example in- identification of mostly service-disrupting bugs. For instance, cludes a “summary” field (Line 25), which typically provides a RESTler discovered 28 bugs causing GitLab [36] service un- brief description of the API’s functionality (similar to the availability, while RestCT [25] identified eight similar issues “description” field). Additionally, there is a “tags” field in Bing Maps [37] and GitLab. USENIX Association 33rd USENIX Security Symposium 741 Table 1: Statistical Data of Known APIs’ Vulnerabilities. Here we list the disclosed API bugs of six common types that we collected from the CVE database. #Bug indicates the number of disclosed bugs. #Ratio indicates the proportion of common functionality in all vulnerable functions. CWE ID Vulnerability Type #Bug Functionality Summary of Vulnerable APIs #Ratio Keywords CWE-434 Unrestricted Upload of File 42 Most of them focus on uploading. 83%(35/42) “upload”,“submit”,“import”, etc. Most of them need to request a remote source, CWE-918 Server-Side Request Forgery 21 81%(17/21) “remote”,“proxy”,“URL”, etc. like Proxy interfaces. They are responsible for handling file through CWE-22 Path Traversal 31 52%(16/31) “path”,“dir”,“file”, etc. a "Path" variable. Most of them are used to set configurations CWE-78 OS Command Injection 55 40%(22/55) “CMD”,“command”,“system”, etc. through commands executed in OS shell. CWE-89 SQL Injection 70 They are responsible for handling SQL database. 53%(37/70) “SQL”,“database”,“select”, etc. CWE-79 Cross-site Scripting 55 They present front-end pages that show text. 35%(19/55) “display”,“content”,“view”, etc. Furthermore, these testing methods often lack a targeted by a specific type of bug belonged to the same functional- strategy and approach based on specific security vulnerabil- ity. These findings confirm the validity of our intuition for ities. Instead, they primarily rely on detecting generic bugs these common vulnerabilities. Additionally, we identified through feedback such as “500 Internal Server Errors” from a group of commonly used strings in specific functionality API endpoints. Consequently, this leads to an indiscriminate as keywords. For example, in an upload interface, we may and inefficient testing approach, with the majority of detected observe frequently used strings such as “upload”, “submit”, bugs being functional issues rather than genuine security vul- and “import” which can be used as keywords to identify cor- nerabilities. responding interfaces from API specifications and support Observation. According to the motivation sample, we ob- subsequent bug detection, as shown in Table 1. It is also im- serve that we can assess the functionality of APIs by exam- portant to note that these six types of vulnerabilities are the ining certain feature strings commonly used to define API mainstream of web vulnerabilities, collectively accounting features in the specification, such as the presence of remote for over 70% of the total [38]. in the current example. Meanwhile, the keyword remote is Our Method. Building upon our observation and verifica- also strongly associated with the type of vulnerability (SSRF). tion result, we propose the core concept of our method, which To validate our observation that the category of vulnerability involves identifying API functions associated with vulnera- present in an API interface is strongly linked to the func- bilities within API specifications and conducting targeted tionality of that interface via some keywords, we gathered security testing on those functions. This approach aims to data from the CVE database on the most common types of achieve vulnerability-oriented API inspection, thereby facili- revealed API vulnerabilities (e.g., Unrestricted Upload and tating the effective discovery of bugs across a wide range of SSRF) and selected vulnerabilities that had been disclosed in RESTful APIs. Different from the previous method, it con- CVE records with specific details, such as the exact location siders the API functionality when identifying and testing for of the bug within the API interface. vulnerabilities. Based on our collected data2 , we first conduct a word fre- quency analysis on the API paths and parameters of API interfaces according to these vulnerabilities’ CVE descrip- 2.3 Challenges and Our Approach tions. Then, combining with expert knowledge (analyze the There are three challenges in realizing vulnerability-oriented root cause of these vulnerabilities), we extract high-frequency API inspection in practice. words that also reflect the functionality of these vulnerable C1: How to efficiently differentiate between various func- APIs, such as "upload" and "submit," and mapped these for tional interfaces in an API and identify functions that may the functionalities (e.g., File Upload function) and vulner- have security vulnerabilities? API documentations usually abilities (e.g., Unrestricted Upload of File). Finally, we try contain a large number of APIs, but only a small portion of to construct a mapping from the vulnerability type to the them may have security vulnerabilities. Hence, it is crucial to corresponding function functionality and its feature strings employ a method that can identify candidate APIs, enabling frequently applied in context. us to enhance testing efficiency and minimize ineffective test Our analysis results are presented in Table 1, which shows requests. that, on average, 57% of vulnerable API interfaces affected C2: How to generate test case sequences that comply with pro- 2 All the vulnerability context information and keywords gathered during tocol states and trigger the vulnerable functional interfaces? this construction process can be found at the link: https://github.com/N API invocations are state-based and often require a series of SSL-SJTU/VoAPI2/CVE-Information.xlsx. pre-request actions (request sequences) to fulfill the execution 742 33rd USENIX Security Symposium USENIX Association /account Test Sequence Generator Test Case /account/email PoC … Generator /avatars/favicon /avatars/image CVE & NVD Expert Analysis Reverse Parameter … Sequence Values /functions … Construction Generation Vulnerability Verifier Path Param Test Case ② Response API Specification Keywords Keywords Sender ① Send ③ Verification ④ Feedback Build API 1. /avatars/favicon tagged_param: url Application Specification Function Vulnerability 2. /avatars/image Bug？ tagged_param: url Parser Mapper … Validation Candidate APIs Server Step 1: Specification Analysis Step 2: Candidate Interface Extraction Step 3: Test Sequence Generation Step 4: Feedback-based Test Figure 3: System Architecture of VOAPI2 . VOAPI2 first analyzes the API specification and identifies semantic keywords associated with potentially weak functions within the API specification. It then integrates a suitable corpus and employs a stateful request sequence that aligns with the execution context of the corresponding functions. By utilizing inspection payloads tailored to different vulnerability types, VOAPI2 can effectively assess these candidate functions for potential vulnerabilities. dependencies of the API, such as providing required input 3 Design parameters, in order to ensure the proper execution of the API. Making standalone API calls is usually considered invalid. Overview. As shown in Figure 3, our approach begins by identifying semantic keywords associated with potentially C3: How to effectively generate valid test cases based on weak functions within the API specification. It then integrates different functional interfaces and corresponding security vul- a suitable corpus and employs a stateful request sequence that nerability types? (1) Different types of vulnerabilities have aligns with the execution context of the corresponding func- distinct principles and require different detection methods. (2) tions. By utilizing inspection payloads tailored to different Overly complex test messages can significantly increase the vulnerability types, we can effectively assess these candidate cost of testing while compromising efficiency and accuracy. functions for potential vulnerabilities. (3) Verifying the presence of vulnerabilities may necessitate additional triggering requests. Therefore, the design of test messages and checkers should strive to encompass a compre- 3.1 Semantic Keyword Collection hensive range of real-world scenarios. VOAPI2 utilizes semantic keywords to identify APIs that To solve these challenges, we design VOAPI2 to detect exhibit specific functionalities. As shown in Table 1, APIs various vulnerabilities in REST APIs effectively. To solve the related to file uploads are often vulnerable to arbitrary file first challenge, we identify and extract the relevant functional uploads. The functionality of an API is typically indicated APIs from the API specifications by utilizing semantic key- by its access path, as demonstrated by /Images/Remote in words (e.g., path keywords and parameter keywords). With a Figure 1. On the other hand, the parameters of an API provide focus on vulnerability analysis, we identify potential vulnera- specific details about its intended function. For instance, a file bility types and their possible locations within these functional upload API is likely to include the keyword “upload” in its APIs. And then, we employ reverse sequence construction access path, while the parameters are expected to involve the and parameter values generation methods, which is expanding keyword “file”. on the techniques used in RESTler and RestCT, to generate In this part, we follow a two-step process to extract API test requests that adhere to the protocol’s state transitions. access path and parameter keywords. First, we collected 544 This ensures the proper context for executing the candidate vulnerabilities related to APIs with publicly available detailed APIs. At last, to address different vulnerability types, we cre- descriptions from sources such as CVE [39] and NVD [40] to ate specific test corpora. By combining the test sequences build a dataset. We then clustered these vulnerabilities based developed for candidate APIs and the marked potential vul- on their CWE (Common Weakness Enumeration) IDs and nerability locations, we incorporate tailored test data into the ranked the CWE IDs according to the number of vulnerabili- corresponding parameters, resulting in comprehensive test ties they encompassed. We selected the six clusters of CWE cases. The presence of potential vulnerabilities in candidate IDs with the highest proportion. Using this dataset as a foun- APIs is assessed by the feedback received from the backend dation, we analyzed the types of API functions associated checker, enabling timely adjustments to the testing strategy with these CWE IDs and created mappings between the CWE when necessary. ID and the corresponding API functionality. Specifically, we USENIX Association 33rd USENIX Security Symposium 743 1 # Endpoint : / Images / Remote , method : Get Table 2: Keywords of API paths and parameters. Here we 2 request = requests . Request ([ list the mappings between Vulnerability Type and API Type 3 primitives . restler_static_string (" GET ") , 4 primitives . restler_basepath ("") , as well as the corresponding number of keywords collected 5 primitives . restler_static_string ("/") , 6 primitives . restler_static_string (" Images ") , in the Semantic Keyword Collection module. 7 primitives . restler_static_string ("/") , #Keywords 8 primitives . restler_static_string (" Remote ") , Vulnerability Type API Type 9 primitives . restler_static_string ("?") , API path API parameter 10 primitives . restler_static_string (" imageUrl =") , 11 primitives . restler_fuzzable_string (" fuzzstring " , quoted = SSRF 10 22 Resource Request APIs False ) , Unrestricted Upload 12 8 File Upload APIs 12 primitives . restler_static_string (" HTTP /1.1\ r\n") , Path Traversal 12 3 Path Processing APIs 13 primitives . restler_static_string (" Accept : application / Command Injection 12 7 System Configuration APIs json \r\n") , 14 primitives . restler_static_string (" Host : localhost \r\n") , SQL Injection 11 4 Database Operation APIs 15 primitives . restler_refreshable_authentication_token (" XSS 15 12 Text Display APIs authentication_token_tag ") , 16 primitives . restler_static_string ("\r\n") , 17 ], 18 requestId ="/ Images / Remote " 19 ) analyzed vulnerabilities within these six CWE clusters and obtained functional descriptions for the vulnerable APIs from Listing 1: Sample of Candidate Interface Extraction. various sources, including source code, API documentation comments, and vulnerability descriptions. We conducted word frequency analysis and leveraged expert experience to identify Listing 1 as an example, we obtained the API access path the frequently occurring API functionalities within these six /Images/Remote (Line 1), the API request method Get (Line CWE clusters. Finally, we established the mappings as shown 1) and the unique parameter of this API imageUrl (Line 10). in Table 2 (i.e., Vulnerability Type and API Type). By applying these patterns, VOAPI2 is able to extract rel- Next, we conducted a statistical analysis of the words found evant information from the grammar file and help identify in the API access paths and parameters, and gathered two dis- the API functions that are associated with potential vulner- tinct clusters of semantic keywords: one for API paths and abilities. VOAPI2 utilizes the semantic keywords generated another for parameters. In detail, we extracted the paths and by the previous module to check for their presence in the parameters from each vulnerable API in the dataset. We then paths and parameters of a given API. Based on this analy- performed word frequency analysis on these paths and param- sis, VOAPI2 categorizes the API functions and assigns them eters, and sought input from human experts to identify the potential vulnerability types (one or more). Our statistical keyword collections. By incorporating the additional insights analysis of past API vulnerabilities indicates that all parame- and expertise from the experts, we refined and expanded the ters could be vulnerable when an API includes path keywords existing keyword collections. This process allowed us to es- but lacks parameter keywords. As a result, we employ the fol- tablish meaningful connections between the keywords, the lowing strategy to extract and classify API functions: (1) APIs corresponding API functionalities, and the vulnerability types. that solely consist of path keywords without any parameter These strings serve as semantic keywords that reflect the func- keywords are identified as candidate APIs, and all parameters tion of the respective category. The number of each type’s of these APIs are marked as test parameters; (2) APIs that keywords is listed in Table 2. contain parameter keywords are also recognized as Candidate APIs, but only the parameters corresponding to the parameter keywords are designated as test parameters. 3.2 Candidate Interface Extraction The extracted candidate APIs are categorized into corre- In this module, we analyze the API specification to identify sponding functional categories based on the keywords, and interface functions that may potentially have vulnerabilities by potential vulnerability types are mapped, which guides the searching for semantic keywords in the API documentation. selection of testing corpus and methods. VOAPI2 categorizes The process involves the following steps. Firstly, VOAPI2 the candidate API functions and their corresponding potential utilizes the RESTler Compile Module to analyze the API vulnerability types as shown in Table 2. specification and generate a grammar file (shown in Listing 1). This grammar file contains information about parameters, 3.3 Test Sequence Generation responses, and the dependencies between individual requests. And then, VOAPI2 parses grammar files based on the This module is designed to create a sequence of API requests following three patterns: (1) the API access path is identi- that accurately reflect the state transitions of the API under fied by locating the string “Endpoint:”; (2) the API request testing. Since API calls often depend on the current state, a method is identified by locating the string “method:”; (3) single request is often inadequate to thoroughly test an API. The API parameters are identified by finding the line above To address this, VOAPI2 employs reverse sequence construc- the restler_fuzzable_string and checking if they are sur- tion techniques to identify other APIs that one candidate API rounded by parentheses “()”. Taking the grammar file in depends on, thereby forming a series of requests that precisely 744 33rd USENIX Security Symposium USENIX Association Algorithm 1 Reverse Sequence Construction Algorithm contains: 1: function G ENERATE _S EQUENCE(candidate_api) P1 =“GET /account” (P represents Producer), 2: S ← candidate_api Pn1 =“GET /database/collections”, 3: i ← -1 4: while True do Pn2 =“POST /database/collections”, 5: producers ← F IND _P RODUCERS(S[i]) Pn3 =“GET /database/collections/{collectionId}”, 6: for producer ∈ producers do Pn4 =“POST /database/collections/{collectionId}/documents”, 7: if I S _VALID _P RODUCER(producer, S) && not and so on. And according to the strategy of the priority I S _D UPLICATE(producer, S) then 8: S.I NSERT(0, producer) system, the producer Pn1 will be removed because it shares 9: if S[i] == S[0] then the same path as Pn2 but has a lower priority. 10: Break CRUD Semantics. CRUD is an acronym that stands for 11: else CREATE, READ, UPDATE, and DELETE. According to the CRUD 12: i = i−1 semantics, a resource (and all its sub-resources) should not 13: return S be accessed before its creation or after its deletion. In reverse sequence construction, the CRUD semantic constraints only capture the state transitions of the API being tested. apply to a pair of producers and consumers. For example, because the consumer’s request method is GET, if there is a producer with a request method of DELETE, we need to remove 3.3.1 Reverse Sequence Construction it from the producer set. We list the reverse sequence construction algorithm Resource Hierarchy. In RESTful APIs, the hierarchical in Algorithm 1 and illustrate the algorithm by using relationships between resources are indicated by the for- GET /database/collections/{collectionId}/documents ward slash “/” in the URLs of the resources. For example, /{documentId}, an API of Appwrite [5]. It initializes a /user/{id} is a direct sub-resource of /user, whereas there sequence set S with the candidate API (candidate_api) and is no direct relationship between /user and /team. We need proceeds to reverse-traverse the set S, starting from the end to remove producers that do not satisfy the resource hierarchy (S[-1] is candidate_api). This process involves identifying relationship. For example, the producer P1 that does not have all producer APIs of the candidate API. For example, for a resource relationship with candidate_api will be removed. the sample API, it searches the responses field in the API After applying these three constraints to filter out documentation for potential producers. It uses string fuzzy invalid producers from the producer set, we are left with matching to match the producers of the parameters (i.e., three valid producers: “POST /database/collections”, collectionId, documentId) of candidate_api. Among these “GET /database/collections/{collectionId}”, “POST producer APIs, only those that meet the producer-consumer /database/collections/{collectionId}/documents”. relationship, CRUD semantic constraints and resource hierarchy constraints (we explain the three key definitions 3.3.2 Parameter Values Generation and their corresponding constraints used in this algorithm in In order to create a concrete request, each input-parameter, the following segments.) are retained and appended to the such as imageUrl (Line 7) in Figure 1, in the sequence needs front end of the sequence set S (Line 6-8). The algorithm to be assigned a value domain. VOAPI2 utilizes one of the then continues the reverse-traversal process, identifying the following strategies to determine the value domain for each producers of the preceding API until it reaches the beginning input-parameter based on RestCT [25]: of the sequence S (Line 9-10). Producer-Consumer Relationship. In the context of the • CONSUMER. This strategy uses the resources from the API testing framework, the Producer-Consumer Relationship producer’s response message (i.e., http response message refers to a rule that governs the execution order of API for one request). If an input-parameter p corresponds to requests based on their input parameters. If the input a resource in the producer’s response message, then the parameter of request A, which can be located in the Path, resource is used as the value domain for p during runtime. Query, Header, Form, or Body, is derived from the response • SPECIFICATION. This strategy uses the values described description of request B using heuristic string matching, in the API specification. If an input-parameter p has an then request B must be executed before request A. In other enum or default field, then the corresponding value is di- words, request B acts as the producer of the input parameter rectly used as the value domain for p. Otherwise, the strat- for request A. In cases where the same input parameter egy searches for all input parameters with the same name as corresponds to multiple producers with the same path but p and selects sample values at random from their example different request methods, a priority system is used to select fields to use as the value domain for p. and add a single producer to the producer set. The priority is determined as follows: POST takes precedence over PUT, GET, • FORMAT. This strategy uses the values from the preset and PATCH. For the sample API, its all potential producers formatting dictionary. If an input-parameter p’s name can USENIX Association 33rd USENIX Security Symposium 745 cases (e.g., evil.jsp and evil.php) tailored to different pro- Table 3: Sample Testing Corpus for Different API Types. gramming languages. For another example, we developed a API Type Sample in Testing Corpus comprehensive set of payloads (e.g., "../"*9+"etc/passwd") Resource Request API http://IP:PORT/ssrf/{0} that attempt to bypass various types of path checks to explore File Upload API evil files (evil.jsp, evil.asp, evil.php, etc) Path Processing API /etc/passwd; C://Windows//win.ini path traversal vulnerabilities in APIs that handle file paths. System Configuration API curl http://IP:PORT/command/{0} More importantly, we design an extensible interface to scale Database Operation API 1" or "1"="1; SQLMap the test corpus. For example, in addition to employing con- Text Display API ventional SQL injection test cases (e.g., 1" or "1"="1), we augment the Test Case Generator by integrating one existing powerful detection tool (i.e., SQLMap [41]) to help determine be matched to a preset formatting dictionary, then the cor- the presence of vulnerabilities with sophisticated corpus. responding value is used as the value domain for p. The VOAPI2 preset formatting dictionary is mainly designed for parameters that have specific formatting requirements, 3.4.2 Feedback-based Vulnerability Verifier such as email (which requires a valid email format) and The Feedback-based Vulnerability Verifier primarily com- password (which requires a length of at least 8 characters). prises two key components: the Test Case Sender and the • SUCCESS. This strategy uses the values from the previous Validation Server. The Test Case Sender is tasked with trans- successful requests (identified by a 200 HTTP status code). mitting test cases to the target API application and receiving If an input-parameter p has the same name as an input- the corresponding responses. It subsequently forwards the parameter from a previous successful request, then the value received response, candidate API information, and testing cor- for that input-parameter is used as the value domain for p. pus data to the Validation Server for in-depth analysis and If the value for the same input-parameter was generated verification. Meanwhile, the Validation Server is responsible using the RANDOM strategy, then the value is mutated for confirming the presence of vulnerabilities. to increase the diversity of test inputs and avoid creating For SSRF, XSS, and Command Injection testing corpus, duplicates. when a vulnerability is present, the vulnerable behavior pro- duced by the testing corpus prompts the API application to • RANDOM. This strategy generates a random value to use send a vulnerability verification request to the Validation as the value domain for an input-parameter p if none of the Server. The Validation Server determines the existence of above strategies can determine a value domain for p. the vulnerability based on whether it receives this request. VOAPI2 applies these strategies in decreasing priority until If the request is received, the Validation Server infers that a value domain is determined for each input-parameter. the vulnerability exists. For the remaining types of testing corpus, the Validation Server determines the existence of vul- nerabilities based on the verification information sent by the 3.4 Feedback-based Testing and Verification Test Case Sender. For example, if the response content of This module consists of two parts: Test Case Generator and an upload API test case contains the term “success” or the Feedback-based Vulnerability Verifier. For each candidate response status code is in the 2xx range, the Validation Server API in the test sequence, Test Case Generator generates valid concludes that the API is vulnerable to Unrestricted Upload test cases with the assistance of the test sequence generation of File with Dangerous Type. Following this, the Validation module. These test cases are then sent to the API application Server further checks if the returned information includes by the Feedback-based Vulnerability Verifier to detect the the path of the uploaded file. If it does, the Validation Server presence of corresponding vulnerabilities. accesses the file and conducts relevant tests to determine if the malicious file can be successfully executed. The Validation Server provides feedback on the verification 3.4.1 Test Case Generator results to the Test Case Generator. When a vulnerability is The test case generator leverages the output of the Test Se- identified, Test Case Generator proceeds to the next test case. quence Generator module to generate test cases that target When Validation Server reports that no vulnerability exists, potential vulnerabilities in the APIs. Specifically, it gener- Test Case Generator makes the following determinations. (1) ates test cases by inserting the testing corpus into the marked The request method in the test case is POST. (2) The testing testing parameter positions. For the unmarked parameters, corpus corresponds to one of the following: SSRF, XSS, or the values assigned by the Test Sequence Generation module Command Injection testing corpus. (3) The testing request are utilized. As shown in Table 3, we collected some test containing the testing corpus has been executed (determined cases and built the testing corpus for different categories of by a 2xx response status code). APIs. For instance, to address variations in server-side archi- When all of these conditions are met, the Test Case Genera- tectures, we gathered and designed malicious file upload test tor infers that a triggering request is required to further verify 746 33rd USENIX Security Symposium USENIX Association the existence of the vulnerability. The triggering request is RQ1 (Vulnerability Detection): How well are the vulnera- sent to further test the existence of the vulnerability. To il- bility identification capability (§5.2) and accuracy (§5.3) of lustrate this point, consider CVE-2023-27161 as an example. VOAPI2 ? Can VOAPI2 discover vulnerabilities in real-world The SSRF vulnerability is located in the /Repositories API APIs? (§5.4) and can be exploited by embedding a payload in the url pa- RQ2 (Efficiency): Can VOAPI2 efficiently generate test rameter using a POST request. However, merely sending the cases and explore operations, and how does it perform com- POST request does not trigger the vulnerability. In such cases, pared with the state-of-the-art tools? (§5.5) an additional triggering request, specifically GET /Packages, RQ3 (Ablation Study): How does the vulnerability-oriented needs to be sent to activate the vulnerable behavior. This trig- strategy of VOAPI2 affect the testing results? (§5.6) gers the API application to send a vulnerability verification request to the Validation Server, which ultimately leads to the 5.1 Experiment Setup discovery of this vulnerability. The triggering requests consist of two types: (1) an API Compared Tools. We compared our prototype system with with the same path and request method as GET, and (2) all several popular vulnerability scanners and state-of-the-art APIs that do not require parameters and have a request method tools in RESTful API testing. It is important to note that these of GET. The selection of these two types of triggering requests tools are designed for different purposes. First, vulnerability is not only aimed at cost savings but also based on historical scanners are primarily used to evaluate web applications. Fur- vulnerability analysis, which has shown a high correlation thermore, these selected vulnerability scanners must support between these types of APIs and triggering vulnerabilities. If API testing by sending mutated requests to these endpoints the vulnerability cannot be triggered through these requests, and reporting potential vulnerabilities based on the response it is recorded as a type of vulnerability that requires manual results. On the other hand, RESTful API testing tools aim to triggering. By introducing a triggering mechanism, VOAPI2 generate better API test cases and discover more bugs (i.e., can more accurately identify API vulnerabilities that require 500 Internal Server Error). However, they are not designed to specific triggering and improve the overall effectiveness of reveal in-depth vulnerabilities. We focus on different aspects vulnerability testing. of VOAPI2 and compare them with corresponding tools to answer the questions we are concerned about, for example, whether our method can find more bugs than vulnerability 4 Implementation scanners. Finally, we select the following two vulnerability scanners and three RESTful API testing tools: We have developed a VOAPI2 prototype system based on RESTler, which consists of 2,700 lines of Python code. The • Zed Attack Proxy (ZAP) [44] is an open-source black-box Semantic Keyword Collection module is implemented based web vulnerability scanner developed by OWASP, primarily on Scrapy [42] and Beautiful Soup [43]. The Candidate Inter- used for web vulnerability assessment and penetration test- face Extraction module is extended from the RESTler’s Com- ing. We leverage ZAP’s resolver to parse API specifications pile module to support parsing API documentation of multiple and inspect API services directly. formats. In the Test Sequence Generation module, the re- verse sequence construction is improved based on RESTler’s • Astra [45] is a scanner specifically designed for vulnerabil- producer search and sequence construction strategy, and the ity scanning of RESTful APIs, testing various vulnerabil- parameter values generation is extended from RestCT’s input- ities by loading payloads into parameters, HTTP headers, parameter value rendering module. In the Feedback-based and other locations. Testing and Verification module, the test case generator is built • RESTler [12] is an open-source black-box RESTful API by modifying the test corpus of RESTler, while the feedback- testing tool developed by Microsoft. It generates stateful based vulnerability verifier is implemented using the socket test cases by inferring producer-consumer dependencies library and by expanding response handling. More impor- between operations and feedback at runtime. tantly, to support various vulnerabilities (e.g., Unrestricted Upload), we have extended the multipart_formdata primitive • RestTestGen [16] uses data dependencies between opera- in RESTler Compile. After all, the current prototype system tions to generate test cases and deploy two distinct oracles supports multiple API documentations adhering to OpenAPI to test RESTful API, that test cases can reveal implementa- v2/v3 and can detect six types of API vulnerabilities. tion defects. • MINER [46] uses length-oriented strategies to generate 5 Evaluation sequence templates and a neural network model to predict key request parameters and provide appropriate parame- We conduct experiments to evaluate the performance of ter values to get a long sequence request in RESTful API VOAPI2 , and our evaluation targets the following questions: testing. USENIX Association 33rd USENIX Security Symposium 747 two vulnerability scanning tools discovered fewer vulnerabili- Table 4: Benchmark Applications. #Endpoint represents ties. The three RESTful API testing tools primarily uncovered the number of endpoints in each Application. #Download bugs that resulted in HTTP 500 errors, with a tiny proportion represents the number that each application is downloaded in being security vulnerabilities, as shown in Figure 4. It can be the Docker hub [47]. observed that VOAPI2 detected vulnerabilities in all seven Applications #Endpoint Version Description #Download applications, with the highest number and variety of vulner- GitLab 358 8.17.0 Code Repository 100M+ abilities. In contrast, the vulnerability scanning tools (i.e., Jellyfin 405 10.7.1 CMS 100M+ ZAP and Astra) did not detect any vulnerabilities in Gitea, Appwrite 95 0.9.3 CMS 5M+ Rbaskets and GitLab. The capabilities of these two tools were Microcks 44 1.17.1 CMS 600K+ Casdoor 121 1.13.0 CMS 20K+ similar, with slight differences observed only in the Appwrite. Gitea 325 1.16.7 Code Repository 20K+ A comparison reveals that VOAPI2 has the strongest ability Rbaskets 22 1.2.3 Web Service 10K+ to identify vulnerabilities compared to ZAP and Astra, par- ticularly in SSRF and XSS vulnerabilities. The other three RESTful API tools discovered only a small number of security Evaluation Benchmarks. We use the following three criteria vulnerabilities in four applications (i.e., Appwrite, Jellyfin, to select real-world RESTful API applications as our bench- Microcks and GitLab), and they required manual payload in- mark for evaluation. (1) Having a complete API specification sertion to validate the security vulnerabilities associated with document that can serve as input for all comparative testing the identified HTTP 500 errors. tools. (2) The application should have APIs covering multiple The three RESTful API testing tools were able to discover orders of magnitude, ranging from small to large-scale. (3) multiple bugs that resulted in HTTP 500 errors when detect- The application should be open-source and widely used in ing targets, as shown in Table 5. The average proportion of different scenarios. Table 4 illustrates the seven applications vulnerabilities among the HTTP 500 errors discovered by the we have chosen as our testing subjects. Based on the number other three tools is very low. Through manual analysis, we of API endpoints, Rbaskets [48], and Microcks [6] are cate- determined that the percentage of vulnerabilities triggered by gorized as small-scale API applications, while Appwrite [5] these errors is only 5.4% (RESTler), 3% (MINER), and 2.5% and Casdoor [4] belong to medium-scale API applications. (RestTestGen), respectively. Because paths with security vul- Gitea [49], Jellyfin [3] and GitLab [36], on the other hand, are nerabilities may not necessarily result in internal errors in considered large-scale API applications. All of these applica- real-world API applications. The mentioned tools randomly tions are easily deployable and supply API services well. modify parameters’ syntax format and content, leading to Evaluation Settings. For the target services, we install them back-end parameter parsing issues rather than posing a sig- with a basic configuration to ensure the proper functioning of nificant security threat. Additionally, incorporating payloads their API services. Moreover, we employ the default settings for all types of vulnerabilities without distinguishing the API for all compared tools and run each with a maximum time functionalities would result in low efficiency, as observed limit (i.e., 5 hours). After each round of testing, we restore the in our subsequent ablation experiments (§5.6). Meanwhile, service environment to ensure the consistency of the target VOAPI2 found fewer instances of HTTP 500 errors, mainly service. We conduct all experiments on a server with 4 CPU due to the targeted nature of VOAPI2 testing, with fewer API cores, 8 GB RAM, and the Ubuntu 20.04 LTS operating sys- paths being tested, which can be observed from the number of tem, which provides adequate resources for conducting the messages. Moreover, VOAPI2 strives to generate messages experiments. In papers such as RESTler, a bug is identified by that comply with grammatical rules to trigger API behavior the reception of a 500 HTTP status code following the exe- and successfully execute the payload that confirms the bug. cution of a request sequence. However, it’s important to note We collected a summary of all the security vulnerabili- that 5xx errors typically refer to server errors during request ties discovered by various tools, including the vulnerability processing, and not all 5xx errors are necessarily related to se- type, vulnerability position, vulnerable endpoint, and vulnera- curity vulnerabilities, such as Denial of Service (DoS). Hence, ble parameters, as shown in Table 6. It can be observed that in our paper, we classify a genuine security vulnerability as a vulnerability-oriented testing methods allow VOAPI2 to dis- bug, distinct from a mere 5xx error. cover corresponding types of security vulnerabilities in APIs with different functionalities. More importantly, VOAPI2 has discovered the highest number of vulnerabilities and the 5.2 Vulnerability Detection (RQ1) widest range of vulnerability types compared to other tools. VOAPI2 identified 26 vulnerabilities, including 7 previously Specifically, VOAPI2 has identified four types of vulnerabili- unknown and 19 known security bugs, across seven API ap- ties, 100% higher than the average of other tools. Meanwhile, plications, encompassing six different types of vulnerabilities. VOAPI2 found 26 vulnerabilities, which was 420% more than All zero-day bugs have been reported to vendors and fixed; the average of other tools. four of them have been assigned CVE IDs. In comparison, the Among these vulnerabilities, our tool uniquely uncovers 748 33rd USENIX Security Symposium USENIX Association SSRF XSS Path Traversal SQL Injection Command Injection Unstricted Upload Appwrite Rbaskets Jellyfin Casdoor Microcks Gitea GitLab 8 8 8 8 8 8 8 7 7 7 7 7 7 7 6 6 6 6 6 6 6 5 5 5 5 5 5 5 4 4 4 4 4 4 4 3 3 3 3 3 3 3 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0 0 0 0 0 0 0 RE 2 Re er M ZA As RE 2 Re er M ZA As RE 2 Re er M ZA As RE 2 Re er M ZA As RE 2 Re er M ZA As RE 2 Re er M ZA As RE 2 Re er M ZA As Vo Vo Vo Vo Vo Vo Vo IN IN IN IN IN IN IN tra tra tra tra tra tra tra stT stT stT stT stT stT stT AP AP AP AP AP AP AP ST P ST P ST P ST P ST P ST P ST P ER n ER n ER n ER n ER n ER n ER n es es es es es es es I I I I I I I l l l l l l l tG tG tG tG tG tG tG e e e e e e e Figure 4: The vulnerabilities and their types uncovered by different tools on evaluation benchmarks . Table 5: Compared with three RESTful API testing tools. #500 means the number of HTTP 500 errors found by tools. #Packet means the number of all packets sent in testing. #Ratio is equal to total #500 divided by total #Packet. Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab Compare #Ratio #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time #500 #Packet Time RESTler 5 88,558 5h 0 165,470 5h 1 297,419 5h 54 2,175 12m07s 19 67,128 5h 0 20,435 5h 32 105,931 5h 0.015% MINER 4 64,944 5h 0 104,352 5h 1 31,668 5h 68 23,002 5h 13 143,830 5h 0 17,422 5h 13 48,267 5h 0.023% RestTestGen 1 9,030 19m49s 0 16,340 34m13s 2 58,200 4h37m 63 71,440 4h43m 15 7,340 8m5s 0 2,140 1m32s 41 57,550 1h52m 0.055% RestTestGen+V 1 6,334 12m42s 0 8,920 13m09s 0 32,550 1h28m 55 48,740 3h57m 11 4,246 5m51s 0 1,550 1m03s 36 43,110 1h33m 0.071% VOAPI2 1 2,123 1m51s 0 6,987 6m27s 0 9,137 7m05s 23 13,578 10m53s 2 509 24s 0 238 13s 3 1,558 5m03s 0.085% 14 bugs that elude detection by other tools. Based on our anal- positive rate. ysis, this phenomenon can be attributed to twofold factors. For path traversal vulnerabilities, ZAP lacks further vali- First, in the case of RESTful API testing tools (e.g., RESTler), dation. It solely bases its determination of a vulnerability’s they often lack a comprehensive testing corpus required for existence on whether the test request returns a 2xx status code. verifying various types of vulnerabilities (e.g., SSRF) within On the other hand, VOAPI2 goes a step further by analyzing their corresponding endpoints. Thus, these tools often fail the response content, checking for the presence of content to detect a vulnerability if it does not lead to a HTTP 500 er- corresponding to the test payload to confirm the existence ror. Second, scanning tools (e.g., ZAP) face challenges in of the vulnerability. For instance, when the test payload is constructing an appropriate sequence encompassing multi- "/etc/passwd", VOAPI2 will match the response content for ple requests that align with the data dependencies existing characteristic strings (e.g., root) to validate the vulnerability. among API endpoints. Thus, these tools fail to identify the re- In terms of VOAPI2 , false positives may occur when check- spective vulnerabilities (e.g., XSS) concealed behind intricate ing a particular XSS (i.e., stored XSS) and the implicit unre- interactions, as exemplified in §5.4. stricted upload vulnerability. In these scenarios, we need to trigger vulnerability manually and check whether vulnerabil- ity exists or not. Moreover, we thoroughly discuss the root 5.3 Accuracy (RQ1) causes and the corresponding improvement ways in §6. We further analyzed the accuracy of VOAPI2 and vulnerabil- ity scanners. All alerts were manually verified to determine if 5.4 Real-world Vulnerabilities (RQ1) they were true vulnerabilities, thereby identifying false pos- itives (FP). The false discovery rate (FDR = FP/(FP+TP)) We applied VOAPI2 to discover security vulnerabilities in was calculated and presented in Table 7. It can be observed real API applications and found various vulnerabilities. As that VOAPI2 has lower false positive rates compared to the shown in Table 6, we identified more types of vulnerabilities scanners, which can be attributed to VOAPI2 ’s vulnerability- compared to scanning tools, especially on XSS and SSRF. oriented strategy and more accurate vulnerability validation The main reason is that our method can generate effective strategy. request sequences that access multiple endpoints in proper For XSS vulnerabilities, ZAP and Astra conduct tests on order, allowing us to trigger deeper vulnerabilities. all APIs and determine the existence of an XSS vulnera- Case Study: XSS. The XSS vulnerability (CVE-2022- bility solely based on the presence of the XSS payload in 2925) was discovered in the Appwrite application. As the response. However, a significant number of these APIs shown in Figure 5, this bug exists in five API end- do not contain display functions. Consequently, the payload points (/teams, /users, /functions, /database/collections, would not be displayed on any particular page, which re- /teams/{teamId}/memberships), and the vulnerable argu- sults in a high incidence of false positives. Correspondingly, ments in these endpoints are "name" marked in blue color. the vulnerability-oriented testing strategy helps VOAPI2 find In our experiments, both ZAP and Astra face significant more API paths related to XSS, which leads to a lower false difficulties. While they may encounter problems in gen- USENIX Association 33rd USENIX Security Symposium 749 Table 6: RESTful API vulnerabilities identified by all tools. For Producer, " indicates this API endpoint has a producer, while % indicates it doesn’t have; For 0-day, " indicates that this vulnerability is a zero-day vulnerability while % indicates it is not; For Vulnerability Identification Tools, " indicates that this tool can identify the vulnerability, " indicates only this tool can identify the vulnerability, while % indicates it cannot. Vulnerability Identification Tools Application Version Path Parameter Type Producer 0-day Bug-IDs VOAPI2 RESTler RestTestGen MINER ZAP Astra 0.9.3 /avatars/favicon url SSRF % " CVE-2023-27159 " % % % " " 0.9.3 /avatars/image url SSRF % " CVE-2023-27159 " % " % " " 0.9.3 /teams name XSS % % CVE-2022-2925 " % % % % " Appwrite 0.9.3 /teams/{teamId}/memberships name XSS " % CVE-2022-2925 " % % % % % 0.9.3 /database/collections name XSS % % CVE-2022-2925 " % % % % % 0.9.3 /functions name XSS % % CVE-2022-2925 " % % % % % 0.9.3 /usrs name XSS % % CVE-2022-2925 " % % % % % Rbaskets 1.2.3 /api/baskets/{name} forward_url SSRF % " CVE-2023-27163 " % % % % % 10.7.1 /Images/Remote imageUrl SSRF % % CVE-2021-29490 " " % " " " 10.7.1 /Items/RemoteSearch/Image imageUrl SSRF % % CVE-2021-29490 " % % % " " 10.7.1 /Items/{itemId}/RemoteImages/Download imageUrl SSRF " % CVE-2021-29490 " % % % % % 10.7.1 /Repositories Url SSRF % " CVE-2023-27161 " % % % % % Jellyfin 10.7.1 /Playlists name XSS % % CVE-2023-23636 " % % % % % 10.7.1 /Repositories name XSS % % CVE-2022-35910 " % % % % % 10.7.1 /Collections name XSS % % CVE-2023-23635 " " % % % % 10.7.1 /Startup/User Name XSS % " 1 unassigned " % % % % % 1.13.0 /api/get-organizations field SQL Injection % % CVE-2022-24124 " % % % " " Casdoor 1.13.0 /api/upload-resource fullFilePath Unrestricted Upload % % CVE-2022-38638 " % % % % % 1.17.1 /jobs repositoryUrl SSRF % " 1 unassigned " " " " % % Microcks 1.17.1 /artifact/download url SSRF % " 1 unassigned " " % " " " 1.16.7 /repos/{owner}/{repo}/contents/{filepath} content Unrestricted Upload " % CVE-2022-1928 " % % % % % Gitea 1.16.7 /repos/{owner}/{repo}/hooks url SSRF " % CVE-2018-15192 " % % % % % 8.17.0 /v3/hooks url SSRF % % CVE-2018-8801 " " % % % % 8.17.0 /v3/projects import_url SSRF % % CVE-2022-0249 " " % % % % GitLab 8.17.0 /v3/projects/{id}/deploy_keys title XSS " % CVE-2022-2230 " % % % % % 8.17.0 /v3/projects/{id}/milestone title XSS " % CVE-2022-1190 " % " % % % Table 7: FDR identified by different tools. Appwrite Rbaskets Jellyfin Casdoor Microcks Gitea GitLab FP Rate 1. /teams: 16. /databases/collections: 2. POST: 18. POST: VOAPI2 4/11 1/2 5/13 2/4 1/3 3/5 3/7 42.22% 3. RequestBody: 19. RequestBody: ZAP 6/8 2/2 9/11 5/6 7/8 5/5 2/2 85.71% 4. name 20. name Astra 5/8 0/0 4/6 3/4 2/3 1/1 1/1 69.57% 5. responses: 21. /users: 6. $id: 22. POST: 7. description: Team ID. 23. RequestBody: 8. /teams/{teamId}/memberships: 24. name 9. POST: 25. /functions: erating appropriate parameter values for some APIs, the 10. parameters: 26. POST: more critical issue is their inherent inability to request 11. description: Team unique ID. 27. RequestBody: 12. in: path 28. name /teams/{teamId}/memberships (Line 8). Because the pres- 13. name: teamId 14. RequestBody: ence of this XSS vulnerability on this endpoint requires ac- 15. name cessing /teams (Line 1) with the POST method before, then parsing the id value (Line 6) from the response, and assign- Figure 5: XSS vulnerabilities in Appwrite. ing it to the path parameter teamId (Line 13). Obviously, ZAP and Astra cannot infer this data dependency and con- struct a suitable sequence that includes these requests. In for testing API endpoints, compared with state-of-the-art contrast, VOAPI2 can extract the context in which the target RESTful testing methods (e.g., RestTestGen); (2) whether API endpoint operates and generate similar request sequences VOAPI2 is more efficient than web scanners (e.g., ZAP) in to discover such vulnerabilities. finding vulnerabilities. Operation Coverage. In this experiment, we used VOAPI2 ’s 5.5 Efficiency (RQ2) Test Sequence Generation (TSG) module to generate a request sequence for each endpoint. We selected three approaches to We check the efficiency of VOAPI2 in two respects: (1) compare the API’s operation coverage with TSG. If a tool whether VOAPI2 can efficiently generate request sequences is able to generate at least one valid request sequence for an 750 33rd USENIX Security Symposium USENIX Association 100 100 100 97.5 96.7 96.7 RESTler MINER 80.0 80.0 RestTestGen 80 75.0 77.3 TSG Coverage % 61.1 62.7 59.1 58.8 60 52.9 50.0 45.3 46.6 44.0 40 37.4 37.3 32.1 30.1 27.4 27.1 28.9 28.6 24.9 21.1 20 11.7 9.8 11.5 11.5 0 Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab Average Figure 6: Compared with RESTful API testing tools on coverage. (§3.2) module from VOAPI2 to ensure all API paths and Table 8: Compared with scanners on testing time consume. parameters are checked with equal priority. We refer to this Appwrite Casdoor Gitea Jellyfin Microcks Rbaskets GitLab #Total modified tool as VOAPI2 -V, which serves as a benchmark for ASTRA 4:30m 18:07m 40:32m 103:28m 8:02m 4:34m 56:16m 235:19m ZAP 10:23m 22:02m 51:27m 88:02m 10:22m 7:57m 182:41m 372:54m comparative analysis against VOAPI2 . ZAP+V 6:39m 8:05m 16:33m 6:11m 1:31m 0:40m 24:07m 63:46m VOAPI2 1:51m 6:27m 7:05m 10:53m 0:24m 0:13m 5:03m 31:05m In testing, no new vulnerabilities were found by VOAPI2 - V. Table 9 shows the time VOAPI2 and VOAPI2 -V need to expose each vulnerability. Obviously, VOAPI2 -V takes API endpoint that results in a 2xx status code response, we several orders of magnitude more time, because VOAPI2 -V consider that endpoint to be covered. must indiscriminately check every endpoint of API. When a Figure 6 exhibits our experimental results, which evidently potentially vulnerable path is located near the end of the spec- demonstrates that TSG performs better than the current state- ification document, VOAPI2 -V takes much longer to reach it. of-the-art RESTful testing tools in terms of generating valid For example, in the case revealed in /Startup/User of Jellyfin, request sequence for target API endpoint. More specifically, VOAPI2 found this vulnerability on this API in 10min30s, TSG achieves the highest endpoint coverage on Appwrite, while VOAPI2 -V takes 495min28s. This is because VOAPI2 Gitea, Microcks, Rbaskets and GitLab. Specifically, the aver- can aim this endpoint directly from a large number of APIs, age coverage of TSG on these products can reach 62.7%, while VOAPI2 -V must test all APIs sequentially. Further- which is more than RESTler’s 37.3%, MINER’s 44.0%, more, VOAPI2 -V tests every parameter of the API using all RestTestGen’s 58.8%. We attribute TSG’s outstanding per- available test payloads, which consumes a significant amount formance to its superior ability to construct dependencies for of time. In contrast, VOAPI2 selects targeted payloads for target API endpoints compared to other tools. For the sample testing the portion parameters marked with directed strategy, API in §3.3.1, we found that none of the tools except VOAPI2 which saves a lot of time. can construct the correct dependency relation for this API. Furthermore, we integrate the keyword shortlisting process Compared With Web Scanners. Table 8 shows time con- into existing tools (i.e., RestTestGen and ZAP). Specifically, sumed by the web vulnerability scanners and VOAPI2 . The these tools are now only applied to candidate APIs extracted test time of VOAPI2 is much less than the other two tools. from VOAPI2 . The integrated models resulting from this Specifically, it took VOAPI2 31min05s to test the seven prod- integration are named RestTestGen+V and ZAP+V. In order ucts, while Astra took 235min19s, and ZAP took 372min54s. to compare the performance of these models, we conduct This is because of VOAPI2 ’s vulnerability-oriented strategy, experiments focusing on efficiency and bug discovery. which just tests candidate API endpoint within targeted pay- In terms of test efficiency, ZAP+V inspects solely the API loads based on the feature of its path and parameters, greatly endpoints that are filtered by keywords. As a result, the testing enhancing testing efficiency. In contrast, tools like ZAP and time is significantly reduced compared to the original ZAP, Astra test parameters and paths indiscriminately within all as showcased in Table 8. This highlights the effectiveness of payloads, resulting in significantly longer testing times. keyword lists when used with ZAP. On the other hand, for RestTestGen+V, we expanded the testing scope to include not 5.6 Ablation Study (RQ3) only the API endpoints filtered by keywords but also those that have data dependencies with them. The testing results To investigate how the vulnerability-oriented strategy of presented in Table 5 demonstrate that these integrated mod- VOAPI2 impacts the result of testing, we perform an ablation els exhibit lower time consumption and send fewer packets study. Thus, we remove the Candidate Interface Extraction compared to their respective original tools. USENIX Association 33rd USENIX Security Symposium 751 Table 9: Compared with VOAPI2 -V on time consume. Time-to-Exposure Can’t execute! Application Bug-IDs Path VOAPI2 VOAPI2 -V 1. /storage/files 2. post: 3. description: Create a new file. 4. operationId: storageCreateFile CVE-2023-27159 /avatars/favicon 2.81s 1min31s 5. requestBody: 6. content: webshell CVE-2023-27159 /avatars/image 2.90s 1min39s 7. multipart/form-data: CVE-2022-2925 /teams 53.42s 26min30s 8. 9. schema: properties: visit 10. file: Appwrite CVE-2022-2925 .../memberships 1min03s 34min36s 11. description: Binary file. 12. type: string CVE-2022-2925 /database/collections 15.28s 17min18s ------------------ Omitted ----------------- CVE-2022-2925 /functions 1min39s 60min16s post CVE-2022-2925 /users 1min30s 48min8s Rbaskets CVE-2023-27163 /api/baskets/{name} 2.08s 38s CVE-2021-29490 /Images/Remote 5.79s 225min46s CVE-2021-29490 /Items/.../Image 7.26s 246min29s CVE-2021-29490 /Items/.../Download 6.68s 228min13s Figure 7: False positive of unrestricted upload. CVE-2023-27161 /Repositories 10.24s 322min38s Jellyfin CVE-2023-23636 /Playlists 9min09s 368min04s CVE-2022-35910 /Repositories 8min57s 322min47s CVE-2023-23635 /Collections 8min25s 35min11s fail to judge these vulnerabilities correctly and require manual 1 unassigned /Startup/User 10min30s 495min28s efforts. To address this issue, we plan to design an extensible CVE-2022-24124 /api/get-organizations 5min04s 43min28s interface of the Feedback-based Vulnerability Verifier (§3.4.2) Casdoor CVE-2022-38638 /api/upload-resource 2min33s 75min50s for specific vulnerabilities. Furthermore, we can continuously Microcks 1 unassigned /jobs 1.90s 11min19s extend vulnerability verification methods, for example, to 1 unassigned /artifact/download 2min06 90min55s heuristically generate upload paths based on the context of CVE-2022-1928 /repos/.../{filepath} 3.61s 94min41s Gitea CVE-2018-15192 /repos/.../hooks 2.49s 36min05s implicit unrestricted upload of dangerous files, thereby en- abling VOAPI2 to discover vulnerabilities more generally CVE-2018-8801 /v3/hooks 1min12s 236min05s CVE-2022-0249 /v3/projects 1min17s 20min17s and automatically. GitLab CVE-2022-2230 /v3/.../deploy_keys 1min49s 53min33s Manual Effort. In identifying and validating vulnerabilities, CVE-2022-1190 /v3/.../milestone 2min36s 145min41s VOAPI2 requires human intervention in two scenarios: im- plicit unrestricted upload vulnerabilities and stored Cross-Site Scripting (XSS). Regarding bug discovery, RestTestGen+V identified fewer HTTP 500 errors, as indicated in Table 5. This can be attributed In the case of unrestricted upload vulnerabilities, VOAPI2 to the fact that the number of candidate APIs is significantly submits a malicious file to check whether the bug exists or not. smaller than the original APIs, and our vulnerability-oriented If the server’s response omits the file’s access path, VOAPI2 strategies are not specifically designed to detect HTTP 500 is unable to automatically retrieve this path, which hinders errors. Furthermore, RestTestGen+V lacks a suitable test cor- further tests to assess whether the file can be executed. We pus that is necessary to trigger potential vulnerabilities in present an example of a false positive of unrestricted upload in the selected APIs. Therefore, it cannot benefit from keyword Figure 7. It can be seen that VOAPI2 automatically deduces shortlisting. Similarly, ZAP+V does not discover more vulner- that the arbitrary file upload API interface permits unrestricted abilities than ZAP because it is unable to construct a suitable uploads of malicious files. However, it lacks the prior knowl- sequence comprising valid requests to satisfy dependencies edge necessary to automatically retrieve the unique access among candidate APIs. paths of these uploaded files, such as identifying the location In conclusion, although keyword selection can enhance of the file access interface or how to obtain random file IDs testing efficiency, it does not lead to better effectiveness when like "651506441776a". This prior knowledge is not standard- directly applied to traditional RESTful API testing tools and ized and tends to vary across different API services, making vulnerability scanners. automatic retrieval of access paths a complex challenge. As a result, manual intervention is currently necessary to visit relevant pages, identify the page containing the file access 6 Discussion interface, and manually test whether the server can parse the malicious file. If the server restricts the parsing of such ma- In this section, we discuss the limitations of VOAPI2 and licious files, the upload vulnerability remains untriggered, explore the improvement direction in the future. leading to a false positive. Scale Ability of Bug Verification. The current state of In terms of stored XSS (also known as persistent XSS), feedback-based vulnerability verification is limited by the VOAPI2 takes a POST request path as a potential XSS vul- concise validation methods based on the testing corpus for nerability after getting a response packet with a 2xx status different bugs. That brings false positives when a bug must be code, which also contains an XSS payload. However, similar triggered interactively, like the stored XSS and implicit upload to the challenges unrestricted upload checking faces, VOAPI2 bug we mentioned before (§5.3). Consequently, VOAPI2 may cannot automatically determine which storage page stores 752 33rd USENIX Security Symposium USENIX Association XSS payloads. Thus, this XSS payload can only be triggered lowing it to reach the "deep" states of the target API service. by manually browsing the relevant pages guided by expert RestTestGen [16] utilizes data dependencies between oper- knowledge. If the XSS behavior is not triggered even after ations to generate test cases, while RESTest [15] considers browsing all relevant pages, it is considered a false positive. dependencies among parameters and generates test cases that Support More Specifications. Currently, VOAPI2 only sup- satisfy specified dependency relationships using constraint ports OpenAPI-formatted API specifications as input, while solving and random input generation. MINER [46] uses a many applications do not provide such documentation and in- neural network model to predict critical request parameters. stead offer API usage instructions in other files (e.g., HTML). However, the aforementioned black-box testing tools all fo- In order to expand VOAPI2 ’s capabilities, we propose to use cus on better parsing API specifications, generating request machine learning or natural language processing methods to sequences that comply with the API interface protocol state, identify and extract API information from these usage instruc- and generating effective test cases that cover more API func- tions. Finally, VOAPI2 can inspect more API applications. tionalities. They aim at the usability of API functions based Machine Learning Method. VOAPI2 classifies API inter- on whether the API service returns HTTP 500 errors but do not faces based on statistics of vulnerability characteristics and specifically target security vulnerabilities. human expertise. Although it is a reliable method to sup- Penetration Techniques. Due to the close similarity in im- port our prototype system in discovering real-world security plementation between the backend of RESTful APIs and tra- vulnerabilities, we believe there is a trend for artificial intelli- ditional web services, researchers aim to expand web penetra- gence technology to replace these expert experiences in the tion techniques to conduct vulnerability detection in RESTful future. For example, advances in natural language processing APIs. Zed Attack Proxy (ZAP) [10] is primarily used for web (e.g., large language model) have demonstrated powerful text vulnerability assessment and penetration testing. Its current analysis capabilities, promising to audit API documents for OpenAPI extension supports parsing API specifications, en- interface classification more effectively. abling web vulnerability testing for API services. Astra [45] is a tool specifically designed to scan vulnerabilities in REST- ful APIs. It verifies common web vulnerabilities by injecting 7 Related Work payloads into parameters, HTTP headers, and other locations. However, these tools have significant limitations that they lack RESTful API Security. The security operations teams the ability to interpret the state of the API protocol. They can of popular API services can discover attacks against their only generate requests for individual API endpoints, limiting services. The Facebook team [10] identified that attackers the scope to test each endpoint separately and preventing the could exploit authorization vulnerabilities in a specific API detection of vulnerabilities in API services composed of mul- endpoint, allowing third-party applications to access users’ tiple endpoints. NAUTILUS [50] incorporates comment poli- private photos. The Twitter team [11] discovered that at- cies into API specifications to handle operation relationships tackers could utilize a specific API endpoint that associates and parameter generation, resulting in meaningful sequences phone numbers with account names to systematically launch of operations and the discovery of corresponding API security widespread attacks and infer the target users’ phone numbers vulnerabilities. However, due to its oversight of vulnerabil- and account names. Researchers note that there are typically ity characteristics, NAUTILUS focuses on a limited range of two categories of bugs in RESTful APIs, including that result vulnerability types, mainly handling injection vulnerabilities. in service unavailability and those related to web security [50]. The former is often caused by syntactically incorrect parame- 8 Conclusion ters in API request data, leading to the backend being unable to process them and resulting in HTTP 500 errors. Most of In summary, we propose VOAPI2 , a novel inspection frame- these bugs typically indicate that the server is unable to han- work, to apply a vulnerability-oriented strategy to inspect dle the current request properly. The latter category arises due RESTful APIs. Based on the insight that the type of vulnera- to similarities between the backend processing of RESTful bility hidden in an API interface is strongly associated with APIs and traditional web services, resulting in web bugs when its functionality, VOAPI2 can directly expose vulnerabilities requests are mishandled. in RESTful APIs. We first track commonly used strings as RESTful Service Testing Methods. Regarding the poten- keywords to identify APIs’ functionality. Then, we generate a tial bugs in RESTful services, researchers have proposed stateful and suitable request sequence to inspect the candidate numerous automated testing methods that can generate se- API function within a targeted payload. Finally, we verify quence requests, assess various endpoints of the API, and whether vulnerabilities exist or not through feedback-based determine if there are any issues based on the service’s re- testing. Our evaluation result shows that VOAPI2 demon- sponses. RESTler [12] is a stateful API black-box testing tech- strates higher efficiency and effectiveness in bug discovery nique that generates stateful test cases by inferring producer- than state-of-the-art tools, including RESTful API testing and consumer dependencies between different API endpoints, al- penetration methods. USENIX Association 33rd USENIX Security Symposium 753 Acknowledgments [14] Stefan Karlsson, Adnan Čaušević, and Daniel Sund- mark. QuickREST: property-based test generation of We thank the anonymous reviewers of this work for their OpenAPI-described RESTful APIs. In International helpful feedback. This research was supported, in part, by Conference on Software Testing, Validation and Verifi- National Natural Science Foundation of China under Grant cation (ICST), pages 131–141, 2020. No. 62372297, Science and Technology Commission of Shanghai Municipality Research Program under Grant No. [15] Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz- 20511102002, National Radio and Television Administration Cortés. RESTest: black-box constraint-based testing of Laboratory Program (TXX20220001ZSB002). RESTful web APIs. In International Conference on Service-Oriented Computing (ICSOC), pages 459–475, 2020. References [16] Emanuele Viglianisi, Michael Dallago, and Mariano [1] Amazon. AWS. https://aws.amazon.com/. Ceccato. RestTestGen: automated black-box testing of RESTful APIs. In International Conference on Soft- [2] Microsoft. Azure. https://azure.microsoft.com/ ware Testing, Validation and Verification (ICST), pages en-us/. 142–152, 2020. [3] Jellyfin. https://jellyfin.org/. [17] Tobias Fertig and Peter Braun. Model-driven testing of RESTful APIs. In International Conference on World [4] Casdoor. https://casdoor.org/. Wide Web: Companion, pages 1497–1502, 2015. [5] Appwrite. https://appwrite.io/. [18] Andrea Arcuri. RESTful API automated test case gener- ation with EvoMaster. ACM Transactions on Software [6] Microcks. https://microcks.io/. Engineering and Methodology (TOSEM), 28(1):1–37, 2019. [7] CVE-2021-3044. https://nvd.nist.gov/vuln/de tail/CVE-2021-3044. [19] Andrea Arcuri. Automated blackbox and whitebox test- ing of RESTful APIs with EvoMaster. IEEE Software, [8] CVE-2019-12643. https://nvd.nist.gov/vuln/de 2020. tail/CVE-2019-12643. [20] Andrea Arcuri and Juan P Galeotti. Handling SQL [9] Representational state transfer. https://en.wikiped databases in automated system test generation. ACM ia.org/wiki/Representational_state_transfe Transactions on Software Engineering and Methodology r. (TOSEM), 29(4):1–31, 2020. [10] Tomer Bar. Notifying our Developer Ecosystem about a [21] Andrea Arcuri and Juan P Galeotti. Enhancing search- Photo API Bug. https://developers.facebook.co based testing with testability transformations for existing m/blog/post/2018/12/14/notifying-our-devel APIs. ACM Transactions on Software Engineering and oper-ecosystem-about-a-photo-api-bug/, 2018. Methodology (TOSEM), 31(1):1–34, 2021. [11] Twitter. An Incident Impacting Your Account Identity. [22] Vaggelis Atlidakis, Roxana Geambasu, Patrice Gode- https://privacy.twitter.com/en/blog/2020/a froid, Marina Polishchuk, and Baishakhi Ray. Pythia: n-incident-impacting-your-account-identity, Grammar-Based Fuzzing of REST APIs with Coverage- 2020. guided Feedback and Learning-based Mutations. arXiv preprint arXiv:2005.11498 [cs.SE], 2020. [12] Vaggelis Atlidakis, Patrice Godefroid, and Marina Pol- [23] S. T. Liu. Coverage guided fuzzing in python-based ishchuk. RESTler: Stateful REST API Fuzzing. In web server. Master’s thesis, Institute of Computer Sci- 2019 IEEE/ACM 41st International Conference on Soft- ence and Engineering, National Chiao Tung University, ware Engineering (ICSE), pages 748–758, Montreal, Hsinchu, Taiwan, 2019. QC, Canada, 2019. IEEE. [24] Runtime application self-protection. https://en.wik [13] Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and ipedia.org/wiki/Runtime_application_self-p Jordi Cabot. Automatic generation of test cases for rotection. REST APIs: a specification-based approach. In Interna- tional Enterprise Distributed Object Computing Confer- [25] Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. ence, pages 181–190, 2018. Combinatorial testing of RESTful APIs. In Proceedings 754 33rd USENIX Security Symposium USENIX Association of the 44th International Conference on Software En- [42] Scrapy. https://scrapy.org/. gineering (ICSE ’22), pages 426–437, New York, NY, USA, 2022. Association for Computing Machinery. [43] Beautiful Soup. https://www.crummy.com/softwar e/BeautifulSoup/. [26] AppSpider. https://www.rapid7.com/products/ appspider. [44] OWASP Zed Attack Proxy. https://www.zaproxy. org/. [27] Qualys Web Application Scanning (WAS). https: [45] ASTRA. https://github.com/flipkart-incubat //www.qualys.com/apps/web-app-scanning/. or/Astra. [28] TnT-Fuzzer. https://github.com/Teebytes/Tn [46] Chenyang Lyu, Jiacheng Xu, Shouling Ji, Xuhong T-Fuzzer. Zhang, Qinying Wang, Binbin Zhao, Gaoning Pan, Wei [29] APIFuzzer. https://github.com/KissPeter/API Cao, Peng Chen, and Raheem Beyah. Miner: A hy- Fuzzer. brid data-driven approach for rest api fuzzing. In 32th USENIX Security Symposium (USENIX Security 23), [30] Patrice Godefroid, Daniel Lehmann, and Marina Pol- 2023. ishchuk. Differential regression testing for REST APIs. In Proceedings of the 29th ACM SIGSOFT International [47] Docker Hub. https://hub.docker.com/. Symposium on Software Testing and Analysis (ISSTA [48] Rbaskets. https://rbaskets.in/. 2020), pages 312–323, New York, NY, USA, 2020. As- sociation for Computing Machinery. [49] Gitea. https://gitea.io/. [31] Y. Luo et al. RestSep: Towards a Test-Oriented Privi- [50] Deng Gelei, Zhang Zhiyi, Li Yuekang, Liu Yi, Zhang lege Partitioning Approach for RESTful APIs. In 2017 Tianwei, Liu Yang, Yu Guo, and Wang Dongjin. Nau- IEEE International Conference on Web Services (ICWS), tilus: Automated restful api vulnerability detection. In pages 548–555, Honolulu, HI, USA, 2017. 32th USENIX Security Symposium (USENIX Security 23), 2023. [32] Y. Luo, H. Zhou, Q. Shen, A. Ruan, and Z. Wu. RestPL: Towards a Request-Oriented Policy Language for Ar- bitrary RESTful APIs. In 2016 IEEE International Conference on Web Services (ICWS), pages 666–671, San Francisco, CA, USA, 2016. [33] Swagger. https://swagger.io/. [34] Vaggelis Atlidakis, Patrice Godefroid, and Marina Pol- ishchuk. Checking Security Properties of Cloud Service REST APIs. In 13th International Conference on Soft- ware Testing, Validation and Verification (ICST), pages 387–397, Porto, Portugal, 2020. IEEE. [35] CVE-2021-29490. https://nvd.nist.gov/vuln/de tail/CVE-2021-29490. [36] Gitlab. https://gitlab.com/. [37] Bing Maps. https://www.bing.com/maps. [38] Cybersecurity Vulnerability Landscape Report. https: //www.h3c.com/cn/d_202303/1796824_30003_0. htm. [39] CVE. https://cve.mitre.org/. [40] NVD. https://nvd.nist.gov/. [41] Bernardo Damele A. G. and Miroslav Stampar. sqlmap. https://github.com/sqlmapproject/sqlmap. USENIX Association 33rd USENIX Security Symposium 755