CrowdStrike donates Falcon MalQuery for rapid YARA hunts to the HA Community

We all know that YARA rules are the pattern matching swiss knife in many ways and have become the de-facto standard when it comes to detection and attributing new malware variants to previously known threats. One great example of this is WannaCry. Within days of that outbreak last year, Google’s Neel Mehta [1] was able to find code similarities between WannaCry code and previously attributed North Korean malware.

Over the past year, the HA community platform has not only been growing at an incredible rate, but also accumulated an amazing collective knowledge (on a side note: thank you, everyone!) due to Falcon Sandbox and many integrations extracting more IOCs than any other community service. Futhermore, to make it easier to help grow the collective community database, we published a new APIv2 with a modern tooling page that acknowledges privacy concerns and is state of the art.

With a growing set of data, having a technology to quickly and accurately search through this data turns a swiss knife to life. Let's imagine we could take any YARA rule or string/binary pattern and scan through petabytes of data within minutes vs. having to wait multiple hours on a batch job to complete? Let's imagine the search results could be downloaded, shared and used to easily determine efficacy and attribution? All within an open and transparent environment that benefits the whole community? Wait, what? This seems too good to be true.

Sometimes dreams come true. ;) Today, we are announcing a revoluationary new search capability to our Hybrid Analysis community platform, which has been implemented as part of a powerful new YARA hunt and binary pattern search capability. Including custom search filters (e.g. date ranges) and efficacy evaluation over petabytes of data. In order to facilitate this type of research by the security community, CrowdStrike has donated Falcon MalQuery, its rapid malware search engine technology, to the community.

Real World YARA Hunting

Let's take a look at how we can utilize the new feature. Let's take a look at the new front page of HA, which contains two new tabs "YARA Search" and "String Search" (for string/binary patterns):-

At this point, you can either navigate directly to the "Advanced Search" form (using the button), drag & drop a YARA rule (from a text file) or paste (CTRL-V) clipboard text containing a YARA rule.

Providing a valid YARA rule takes the user to the following search form/page:-

At this page, it is possible to "refine" the search through a variety of criterias, e.g. the file type(s) to require (in the search result set), as well as date and file size ranges.

After performing the hunt (in this case a RAT of the Lazarus Group), a list of matched HA Community Files is presented, including links to the respective reports and a collective IOC download (via CSV):-

Happy hunting!

Source: https://www.hybrid-analysis.com/string-search/results/3530edadd82a867e3a35f4b26d8f85b5e82eb67d4b759cc1f77d2f499571cf59

[1] https://twitter.com/neelmehta/status/864164081116225536?s=21

Changelog Q4 2015 - Q1 2016 (distilled)

We've been so busy improving VxStream Sandbox and the surrounding technology that we have been having a bit of an on-off relationship with our blog. Today we wanted to catch up a bit and let everyone know what we have been up to, who have not been following extremely closely. Besides visible changes, there has also been a lot of improvements going on in the backend: for example, we have been working heavily on automating the deployment of the standalone system, as it is our vision to enable a fully unattended installation of the entire system (i.e. not only the server/host environment, but also the installation and configuration of Windows, including third-party dependencies). Currently, taking a server from scratch, it is possible to setup the entire system in less than two hours to a state that allows a first end-to-end analysis. Either way, in the following we will try to touch on the most notable improvements and additions that were implemented over the past five months.

New Input Formats

As we see VxStream Sandbox as an "engine" that should retain a high level of usability, but more importantly support as many input and common output formats as possible. That means supporting also uncommon file formats, such as Outlook msg or processing MIME types, which we added just recently. In case you wonder what we support exactly, you can always find an up-to-date list in the FAQ, a good resource to answer some other questions as well. Currently, the supported formats are: any kind of PE (.exe, .scr, .pif, .dll, .com, .cpl, etc.), Office (.doc, .docx, .ppt, .pptx, .xls, .xlsx, .rtf), PDF, APK, executable JAR, Windows Shortcut (.lnk), Windows Help (.chm), Javascript (.js), Shockwave Flash (.swf), Powershell (.ps1, .psd1, .psm1), MIME RFC 822 (*.eml) and Outlook *.msg files.

 

New Output Formats 

Similar improvements took place to the list of supported output formats. Most notably is the addition of MAEC 4.1 and OpenIOC 1.1, which can now be generated and used to share indicators. One has to say that though that, unfortunately, Mandiant (the consortium behind the OpenIOC format) surprisingly do not support their own latest OpenIOC 1.1 format with their core tools, foremost the OpenIOC editor in its latest version 2.2 (published end of 2012 while the latest format definition was released in 2014). So our decision to go for the latest definition was in parts a technical one, but more of a business one at the end of the day. We hope to create more incentives for the FireEye owned company to synchronize their toolsets with their own standards. As the transition from 1.0 to 1.1 is more cosmetic than fundamental, this would probably be a good weekend project for the next intern, but enough of this rant. What else did we change? Well, other output related improvements were internal ones focusing around the HTTP based API of VxStream Sandbox that we plan on possibly making available to the public later this year.

More 'Hybrid Analysis' Integration (Example)

We made a tweet about it end of January (we often tweet features or things we find interesting on our Twitter account, which is probably a better chronlogy of what we have done than this blogpost) about a new integration we added: whenever we find a URL or IP address (i.e. something that may be related to network behavior), a 'Memory Forensics' sub-section in the 'Network Data' area of the report will appear:

While this is just a "tiny" feature, it does help make possible network intelligence available to the analyst (as the network endpoints may or may not have been used for live communication, as they are based purely on memory dump analysis) and give entrypoints for a deeper analysis at the same time. Please note that in the full version of VxStream Sandbox you can download associated memory dumps for a follow-up analysis.

Improved Static Analysis

We continued to improve a variety of static analysis techniques that we apply to input samples such as PDF file(s) and the associated URL extraction and evaluation. More notably, we improved our proprietary Javascript and VBA/VBS deobfuscator engines and also added a proof of concept Dridex config extractor (built upon initial work by malware.lu, much kudos from here):

View Sample Report

Improved False Positives

We noticed that one major challenge for any sandbox system that does not have a huge whitelist in the backend is to reduce the false positive ratio. For example, often installers (such as Skype) will behave in a way that is very difficult to put apart from malware without a reputation database or whitelists. Installers are often packed, drop executable files, spawn child processes, persist themselves, possibly download files from abnormal ports, may connect to a wide range of servers, try to install system services and so on. The same goes for some of the latest document "launcher applications", such as Acrobat XI, which come with their own sandboxing technology and make it difficult to put aside whitenoise, as the propriertary code behaves quite 'naughty' itself (e.g. patching the own process, and so on). Thankfully, VxStream integrates with NSRL and can read the "isgoodware" flag of VirusTotal, so we got a handle on many false positives through improved verdict adjustments regardless of the actual behavior. Over the past months and even weeks, we have been continuing to tweek the system in this respect, something that is not easily visible from the online reports of the free webservice. For example, we will lower the verdict from "malicious" to "suspicious" if we find a variety of artifacts that we learned are combined a strong statement of belonging to benign artifacts. A possible example: if a file is clean on a multiscan AV engine, has a valid certificate, drops files that are explicitly flagged as benign, executed without crashing and showed reasonable behavior, then we could apply such verdict downgrading. For us the hurdle for a downgrade has to be very high, as we believe a false negative to be far worse than a false positive.

Another addition is the "no verdict" verdict that some may have noticed, which essentially means "there is too little data to make a reliable determination of whether or not the file is benign or malicious". We believe industry should be open and fair about verdicts and let's be honest: we have all encountered a situation in life where we simply don't know. Instead of pretending we have a good, reliable opinion of a file (and sometimes it is very difficult), we believe it is only fair to communicate that to the user and processing system. This also has the benefit that the system user may decide on a more restrictive or casual policy on those cases. Unfortunately, we cannot disclose under which conditions the "no verdict" appears.

Metadefender OPSWAT Integration

While on the one hand we are trying to make VxStream Sandbox a great "engine" for processing any file and making the data available in any format, we also believe it is equally important to integrate and support interfaces of a variety of excellent third party tools, such as VirusTotal - and now also - OPSWAT Metadefender (formerely 'Metascan'). Giving customers the ability to utilize a purchased API key from one of these vendors is a great oportunity to add yet another layer to your defenses.

URL analysis

This was a major feature on our roadmap for quite some time and while it is always a work in progress (similar to Google's "Forever Beta"), we have made the URL analysis feature available on the front page of the webservice to give it the best stress test you can have, which is being available in the wild:

While the URL analysis is sitll in its baby shoes, there is some basic exploit detection through browser emulation, file extraction, browser process monitoring and we implemented common processing of files extracted in the context of a webpage (VirusTotal lookup, YARA signature matching) as-if it were a dropped file on a regular analysis.

Note: we had to take the feature down on the free webservice, as we have been getting massive amounts of submissions and accesses to our webservice, which is a single server at this point. The URL analysis is available to all private cloud customers, which are also hosted on another server. More information here: https://www.vxstream-sandbox.com/

Final Words

 

In this blogpost we highlighted a few of the new supported file formats, integrations and features added over the past months. As Payload Security consists of 90% developers and 10% marketing/sales, we often move faster than we can communicate things in fancy flyers or conference 'silver partner' stands (and you will most likely not see us at any conference anytime soon). The outlined changes only touched on a subset: there has been many other improvements, hidden ones (such as anti-VM detection technology improvements or deep packet inspection), things like MISP pathway normalization, many new indicators, code refactorings, optimizations, tons of bug fixes (one bug every 1k LOC, remember?) and so on. We hope you will continue to enjoy our free service at https://www.hybrid-analysis.com/ and feel free to follow us on Twitter for a more frequent news update.

Follow @PayloadSecurity

Latest Updates of VxStream Sandbox and the Malware Analysis Service at Hybrid-Analysis.com

A previous blogpost published at the beginning of February outlined some of the new features that were added to our online malware service. We have added quite a lot of functionality since then and think it is a good idea to post a brief summary of what that is exactly to keep our readers and users up-to-date.

 

Updated Anti-VM Technology

After Pafish v0.4 (a benchmarking tool that implements common VM detection methods) was released earlier this year, we updated our anti-vm technology to be up-to-date and made a small benchmark of some popular malware analysis services at the same time. Today, Pafish v0.5 was released and we will start working on our anti-vm technology in the coming weeks and keep you updated on any progress.

 

Improved Searching Capabilities

We improved the webservice search and added some more advanced search options. On the previous version, you were able to search by filename, MD5 or SHA256 hash. Now, you can also search for a virus family name, all reports that contacted a specific host IP address or domain. Examples:

• host:95.181.53.78
• domain:checkip.dyndns.org
• vxfamily:upatre

Please note: if only one result is returned by the search, you are automatically redirected to the report. Also, the vxfamily search is a substring search and applies only to the VxStream determined virus family name. All search results are limited to at most 100.

Also, some of the new searching capabilities were integrated into all online reports with direct links, so you can continue navigating to other reports by clicking the virus family name or quickly find other reports with common network destinations (see the following image).

 

Updated VBA Macro Parsing

As we had been getting more and more uploads of Word files and malicious XML files (and not all of them triggered or showed outgoing network traffic), we spent some time and added a small VBA "de-obfuscating" engine that helps extracting C2 IPs regardless of the runtime behavior. We made a blogpost about it last week that received good feedback and is showing some good results so far. After we published the blogpost, Philippe Lagadec announced that he is working on a generic engine that does the same and more - so we are looking forward to that development and will keep you updated on any progress.

 

Other updates not mentioned anywhere

Of course, we also make updates that are not published as part of blogposts or mentioned in the FAQ page of the service, because it would take too much time and not everything is really significant. Some of these updates over the past week included:

• we added new YARA signatures that run on all input samples (we have ~600 online right now)
• we have been adding more generic behavior signatures (we have ~215 online right now)
• we added a webservice statistics page to clean up the front page, which tells you the current status of the number of signatures loaded by the system
• we added support for MIME types (i.e. you can upload a MIME type and the service will "unmime" it and analyze a valid file, if it is embedded)
• we added "environment groups" (multiple systems) that can be selected from if you upload a file
• we added some Windows 8.1 VMs
• we added the ability to "not share" a sample when submitting (it is not available for download and not uploaded to VirusTotal, if unknown)
• we added a download for strings detected in-memory
• we added shellcode streams that are extracted from memory written to foreign processes
• we brushed up the visuals a bit, especially the submissions list that contains a lot more information now

.. and a few other minor things that should not be mentioned here.