Analyzing Honeypot SSH Traffic
A few months ago, I set up a DShield honeypot on an AWS free tier EC2 after learning about DShield honeypots at a SANS class I took in July of 2022 (see my previous article on how to set up the honeypot in AWS). Once I set it up, I watched all the logs flow in from the firewall, SSH, and web honeypots incorporated into DShield. If you choose to upload the data to the SANS Internet Storm Center, they give you some nice pre-made graphs and charts to help visualize the data. But after a while, I decided I wanted to do a more in-depth look at the data that was coming in. So this article is going to be my first shot at analyzing the traffic hitting the SSH honeypot from 6/4–6/11. Here are the graphs and charts for 6/11:
Although not labeled, the X axis of the top line chart is time, with the left most value being 5/13 and the right most being 6/11. The sharp drop off about three fourths of the way through the line chart is due to me shutting off the EC2 instance for a few days so as not to exceed the free tier license. However, I do find it odd that the number of reports isn’t back up to normal after a few days. I’m unsure of why that is. I do update the instance and honeypot at the beginning of each month, so perhaps it could be related to that, vis-a-vis the instance having a new IP and DNS name. I will not be going into much detail about the graphs as I feel they speak for themselves, and I will be taking a closer look at the IPs, usernames, and passwords later on.
You can also download CSVs of the data in the form of pre-canned reports. Here is one that summarizes the passwords, source IPs, and usernames for 6/11:
Also available is a table of the number of reports that came in per day, and the number of unique passwords and IPs seen in those reports over the past month:
IP Addresses
First let’s take a closer look at the IP address. I decide to slice and dice the data in three different ways:
- The number of SSH attempts by IP per day.
- The dates an IP was seen as active.
- A total count of the number of SSH attempts by unique IP.
To visualize the first way of displaying the data, I used a Pivot table in Excel to show the each IP as a subset of the date it was seen on. The IPs can be further drilled into to see the username and associated passwords it attempted to SSH in with:
For some reason, I was unable to pull raw data from 6/9 and 6/10. As we can see, it looks like the majority of the SSH attempts took place between 6/6 and 6/8. 6/8 not only had the highest count of SSH attempts, but also the most unique IP addresses.
Next, I was curious if any of those IPs were repeat offenders. So, I created another Pivot table that essentially swapped the ordering of the date and IP in order to see the dates each IP was active on:
I was expecting to see multiple IPs active across multiple days, but it appears most ceased their SSH attempts after one day. The notable exception being 170.64.138.72 which was active for three days. Interestingly, those three days were also the ones with the most amount of incoming attempts.
Finally, I wanted to know which IPs were the most active. A simple sorted Pivot table will display this what we want to know:
As we can see, 170.64.138.72 is by far the naughtiest. Let’s see if we can find out more about a few of these IP. Here are the ones I decided to take a closer look at:
- 170.64.138.72: This was the IP that was most often seen and had the most number of SSH attempts, so any obvious choice for further analysis. Here is an example of the usernames it used on 6/6:
- 171.244.53.224: I picked this IP because it was also one of the top 5 contributors, and besides 170.64.138.72, it tried to use one of the widest varieties of usernames. Therefore, it seemed less “generic” to me, and that is was more likely that some actual thought had been put into targeting certain services. By that I mean, it wasn’t only trying “root” or “admin” with a bunch of common passwords. I mention services because based on the usernames being used by all of the IPs in question, most of the accounts seem like default or service accounts and not those of a human user.
- 35.203.210.113: I picked this IP because, for what ever reason, the username and password it used looked like parts of an HTTP request. The password even contained another IP, so I thought that would be interesting to look into.
170.64.138.72
Doing a basic lookup, we can see that it appears be a mail server with the hostname “mail3.g24.pair.com”. The server looks like its being based out of a DigitalOcean datacenter in Australia.
However, a DNS lookup for that hostname shows that it resolves to a different IP:
I’m not sure why that is the case, as I would think that DNS lookup would return our IP address. Interestingly, I did a reverse DNS look up on 170.64.138.72 a few days later and no hostname was returned.
The subdomain g24.pair.com didn’t have any DNS records, but the domain “pair.com” did:
“pair.com” appears to be a company offering web hosting services:
In VirusTotal, 10 vendors flagged the IP as malicious:
I was looking around and I saw there were some detections under Passive DNS Replication. I wasn’t previously familiar with the concept, and I haven’t been able to find any explanations that made the idea click for me, but the one that VirusTotal gave seemed straightforward enough. It seems that VirusTotal was able to find historical DNS records from a few months ago indicating that the two subdomains of “aytashpo.ru” seen in the above photo resolved to the IP address we’re looking at. “53desire.aytashpo.ru” and “aytashpo.ru” were also flagged as being malicious:
Circling back to the hostname and domain for the IP, “mail3.g24.pair.com” and “pair.com” came up as mostly clean, but “mail3.g24.pair.com” seemed to have some malicious files that reached out to the it upon execution:
We know the IP associated with “mail3.g24.pair.com” was previously associated with subdomains of “aytashpo.ru”, which looked like they were used for C2 traffic. But as these malicious files apparently reached out specifically to “mail3.g24.pair.com”, I’m unsure why the hostname wasn’t flagged malicious as well. I checked out the first file, and it looks like some kind of malicious dropper trojan for Windows. The engines seemed to think it was either a variant of Jaik, Kryptik, or Redline malware.
171.244.53.224
Doing another basic lookup, we can see 171.244.53.224 is based out of Ha Noi. I couldn’t find a hostname associated with it via a reverse DNS lookup.
VirusTotal definitely flagged it as malicious, but other than that there wasn’t much interesting about this IP address.
I looked up the Kubernetes Ingress Controller Fake Certificate (from the Subject of the HTTPS certificate) and I got a bunch articles for troubleshooting trying to replace that certificate with another one. It seems the certificate is a self-signed one that comes as a default when trying to set up Ingress for Kubernetes. I also tried throwing the IP into GreyNoise, but it didn’t give me much else to go off of:
Overall, the IP wasn’t as interesting as I had hoped it would be. I was disappointed it didn’t lead to anything else like the previous IP did.
35.203.210.113
A basic look up shows this IP is based out of London.
After doing some research, it appears the syntax of the hostname indicates it is related to a GCP Compute Engine (how GCP names their compute instances was previously unknown to me):
There were no DNS records for “bc.googleusercontent.com” or “googleusercontent.com” (in retrospect, not surprising since its a CDN):
VirusTotal flagged it as malicious and further confirmed that it belong to GCP, but other than that didn’t provide much info:
GreyNoise determined it was benign, and the User Agent indicated that it belonged to Expanse, a part of Palo Alto Networks, which is used for automated Attack Surface Management.
Some research on “googleusercontent.com” reveals it is some sort of CDN, possible for image files. I also came across an interesting article that talked about threat actors hosting images containing malicious code embedded within the image itself on Google’s CDN “googleusercontent.com”.
The password field of the SSH attempt contained what appeared to be the contents of the Host header of an HTTP request. The IP was indeed that of my honeypot (which is why its redacted), and it was trying to get at port 2323 which is used as an alternative to Telnet port 23. I’m not sure how that all ended up as the password, I’m guessing it was some error in how the honeypot was handling traffic.
This IP ended up having a few more interesting quirks than the last one, and I learned some new thing about “googleusercontent.com” along the way.
Usernames
Here is a list of the SSH usernames attempted:
+----------------------------+-------------------+
| Row Labels | Count of Username |
+----------------------------+-------------------+
| root | 282 |
| admin | 42 |
| ubuntu | 26 |
| user | 17 |
| hp | 15 |
| git | 12 |
| guest | 11 |
| dolphinscheduler | 10 |
| dell | 8 |
| hadoop | 7 |
| asus | 7 |
| centos | 7 |
| oracle | 6 |
| testuser | 6 |
| asterisk | 6 |
| zookeeper | 6 |
| mysql | 6 |
| kafka | 6 |
| ubnt | 5 |
| ps | 5 |
| telnet | 5 |
| postgres | 5 |
| miner | 5 |
| ansible | 4 |
| gns3 | 4 |
| samba | 4 |
| jenkins | 4 |
| lighthouse | 4 |
| usr | 3 |
| share | 3 |
| zabbix | 3 |
| gf | 3 |
| test | 3 |
| amax | 3 |
| wjy | 3 |
| gitlab-runner | 3 |
| server | 3 |
| gpadmin | 3 |
| tao | 3 |
| a | 3 |
| elk | 3 |
| aaaa | 3 |
| vyos | 3 |
| abc | 3 |
| xxj | 3 |
| chen | 3 |
| ftpuser | 3 |
| kettle | 3 |
| sftp | 3 |
| cheng | 3 |
| star | 3 |
| minerstat | 3 |
| Administrator | 3 |
| mongodb | 3 |
| tomcat | 3 |
| cxc | 3 |
| es | 3 |
| nexus | 3 |
| usuario | 3 |
| czq | 3 |
| wang | 3 |
| peter | 3 |
| www | 3 |
| pi | 3 |
| yang | 3 |
| dev | 3 |
| ftp | 3 |
| dmdba | 3 |
| docker | 3 |
| sun | 2 |
| 888888 | 2 |
| webadmin | 2 |
| 666666 | 2 |
| telecomadmin | 2 |
| elasticsearch | 2 |
| supervisor | 2 |
| zxfang | 2 |
| support | 2 |
| liwq | 2 |
| teamspeak | 2 |
| minecraft | 2 |
| awsgui | 2 |
| sonar | 2 |
| elastic | 2 |
| jupyter | 2 |
| kubernetes | 2 |
| Contact: <sip:nm@nm> | 1 |
| Call-ID: 50000 | 1 |
| mother | 1 |
| tech | 1 |
| service | 1 |
| OPTIONS sip:nm SIP/2.0 | 1 |
| CUAdmin | 1 |
| fa | 1 |
| moxa | 1 |
| orangepi | 1 |
| Max-Forwards: 70 | 1 |
| From: <sip:nm@nm>;tag=root | 1 |
| CMCCAdmin | 1 |
| debianuser | 1 |
| craft | 1 |
| pineadmin | 1 |
| steam | 1 |
| admin1 | 1 |
| zjw | 1 |
| GET / HTTP/1.1 | 1 |
| b'0„-c„$' | 1 |
| ansadmin | 1 |
| akash | 1 |
| Grand Total | 705 |
+----------------------------+-------------------+Most of the usernames look like pretty standard, and easily guessable for that matter, account names. They strike me as probably being service accounts of some sort or another. I was discussing how to go about analyzing these usernames with a colleague, and he suggested that, while the requirements for each service to function probably vary widely, we must consider the question of how many of these services actually require SSH to be enabled. He further pointed out that if a service doesn’t require SSH but it is still enabled, it’s plausible to suggest that such a misconfiguration indicates that the password hygiene and policies are likely poorly implemented as well. One can assume that this pattern of poor configuration is what the threat actors are hoping for. For example, part of me was surprised to see how much more prevalent the “root” username was compared to the others. I certainly didn’t think it would be tried nearly 7 times more often than the next most common username. My reasoning being that I would think that root login would be disabled in most cases. But I suppose its never a good idea to underestimate the number legacy or misconfigured systems are sitting out on the internet at any given time.
Furthermore, I also wondered whether or not the frequency with which some of these usernames show up can be attributed to either recent or highly exploitable vulnerabilities associated with that particular service. The median number of times a username was tried is 3 in our data set. I thought median would be a more appropriate statistic since the mean (6.47 in this case) is more sensitive to skew and outliers, which our data set seems to have. So, taking a look at a few of the potential services whose corresponding username were tried more than 3 times, we can look to see if there are any vulnerabilities for that service that could maybe explain why they were target more often. Given the relatively unsophisticated nature of the scans hitting my honeypot, I tried to look for vulnerabilities over the last year that would have the highest impact with the lowest effort for the attacker (think along the lines of the headline grabbing critical CVEs). Let’s take a look at a few of them:
- git: The git username was tried 12 times. After looking around for recent vulnerabilities, it seems git had quite a few. CVE-2023-41903 and CVE-2023-23521 from January stood out as being particularly nasty, as they are both critical RCEs (read more about them here).
- dolphinscheduler: DolphinScheduler “is a distributed and extensible open-source workflow orchestration platform with powerful DAG visual interfaces”. I found a critical RCE from late 2022, CVE-2022–45875, and critical command injection vulnerability also from late last year CVE-2022–45462.
- zabbix: Zabbix is on open-source tool for monitoring IT infrastructure. It looks like there was a critical authentication related vulnerability, CVE-2022–23131, that came out early last year.
There are several caveats that I want to acknowledged about the whole premise that services with recent critical vulnerabilities could lead to the higher SSH username attempts we saw. The first being that correlation does not equal causation. I don’t think we can definitively prove, based on what we’ve seen, a cause and effect relationship between the frequency of certain usernames being used in SSH attempts and the existence of critical vulnerabilities in services associated with those usernames. But I do believe there is some element of truth to the idea as it makes sense from an attacker perspective. If I know there are services out there with critical flaws that I can easily exploit, then, depending on my goals, that’s what I’m going to go after since I want the most bang for my buck. Second, none of the vulnerabilities I mentioned seem to have anything to do with SSH. CVE-2022–23131, for example, seems like it could be exploited without requiring any access to a user account via SSH. So I think it goes back to what my colleague was saying. If I was able to exploit some critical vulnerability that should have been mitigated or patched but wasn’t, then it seems like odds are good I could somehow abuse SSH access. Or on the other side, if I could abuse SSH access, then perhaps the odds are good that certain services are unpatched. This all involves a lot of assumptions, but once again, as the attacker you have to take in all the information available to you in order to make educated guesses about the security posture of your target so you can plan out the best route of attack.
Passwords
Here is a list of all of our passwords:
+--------------------------------+-------------------+
| Row Labels | Count of Password |
+--------------------------------+-------------------+
| 123456 | 112 |
| 123 | 19 |
| root | 18 |
| password | 16 |
| abc@123 | 12 |
| admin | 12 |
| 12345 | 11 |
| 1234 | 10 |
| 1 | 9 |
| ubuntu | 8 |
| admin123 | 8 |
| P@ssw0rd | 7 |
| 0 | 7 |
| 12345678 | 6 |
| root@123 | 6 |
| user | 6 |
| Huawei@123 | 6 |
| centos | 6 |
| mmpOS | 5 |
| qwer1234 | 5 |
| postgres | 5 |
| guest | 5 |
| 1234qwer | 5 |
| oracle123 | 5 |
| dolphinscheduler | 4 |
| root123 | 4 |
| 666666 | 4 |
| loler123 | 4 |
| ubnt | 4 |
| 123456789 | 4 |
| 1qaz@WSX | 4 |
| 1111 | 4 |
| 888888 | 4 |
| ansible | 4 |
| telnet | 4 |
| pass | 4 |
| lighthouse | 4 |
| Aa123456 | 4 |
| 7ujMko0admin | 4 |
| hadoop | 4 |
| nexus | 3 |
| aaaa | 3 |
| root1234 | 3 |
| 123123123 | 3 |
| www | 3 |
| 112233 | 3 |
| 1qaz@WSX3edc | 3 |
| asterisk | 3 |
| 54321 | 3 |
| 1223334444 | 3 |
| user123 | 3 |
| dev | 3 |
| zookeeper | 3 |
| dmdba | 3 |
| 123321 | 3 |
| 1234567890 | 3 |
| qwerty | 3 |
| dolphinscheduler123 | 3 |
| sftp | 3 |
| dreambox | 3 |
| testuser | 3 |
| elk123 | 3 |
| ubuntu123 | 3 |
| es | 3 |
| vyos | 3 |
| ftp | 3 |
| zabbix | 3 |
| ftpuser | 3 |
| mysql | 3 |
| gf | 3 |
| 123456zh | 3 |
| git | 3 |
| 1122 | 3 |
| git123 | 3 |
| 1qaz2wsx3edc | 3 |
| gpadmin | 3 |
| a | 3 |
| 1.11122E+15 | 3 |
| server | 3 |
| hp | 3 |
| share | 3 |
| hp123 | 3 |
| test | 3 |
| Huawei12#$ | 3 |
| tomcat | 3 |
| !@#123qwe | 3 |
| 111111 | 3 |
| kafka | 3 |
| 111222333 | 3 |
| kettle | 3 |
| usuario | 3 |
| klv123 | 3 |
| wjy | 3 |
| !QAZ@WSX | 3 |
| xc3511 | 3 |
| Lt111111 | 3 |
| zh123456 | 3 |
| mongodb | 3 |
| admin@123 | 3 |
| msos | 3 |
| webadmin | 2 |
| meinsm | 2 |
| ikwb | 2 |
| realtek | 2 |
| dell | 2 |
| rock | 2 |
| xmhdipc | 2 |
| 5up | 2 |
| 123.com | 2 |
| 13579 | 2 |
| 123456.cn | 2 |
| root1 | 2 |
| hi3518 | 2 |
| jupyter | 2 |
| 163.com | 2 |
| awsgui | 2 |
| qweasdzxc | 2 |
| Root123456 | 2 |
| zxfang | 2 |
| root1337 | 2 |
| gns3 | 2 |
| root2022 | 2 |
| minecraft | 2 |
| root666 | 2 |
| elasticsearch | 2 |
| root777 | 2 |
| 1/2/03 | 2 |
| samba | 2 |
| admin1234 | 2 |
| aquario | 2 |
| eve | 2 |
| default | 2 |
| sun | 2 |
| kubernetes | 2 |
| founder88 | 2 |
| support | 2 |
| zhongxing | 2 |
| system | 2 |
| Zte521 | 2 |
| teamspeak | 2 |
| asus123 | 2 |
| elastic | 2 |
| juantech | 2 |
| 123123 | 1 |
| 1111111 | 1 |
| oracle | 1 |
| ivdev | 1 |
| qazxsw | 1 |
| jenkins | 1 |
| craft | 1 |
| 141388 | 1 |
| CUAdmin | 1 |
| 1q2w3e | 1 |
| Password123 | 1 |
| jvbzd | 1 |
| zlxx. | 1 |
| admintelecom | 1 |
| nE7jA%5m | 1 |
| akash | 1 |
| ubnt1 | 1 |
| anko | 1 |
| orangepi | 1 |
| klv1234 | 1 |
| Via: SIP/2.0/TCP nm;branch=foo | 1 |
| root2023 | 1 |
| Win1doW$ | 1 |
| root321 | 1 |
| pi | 1 |
| ansadmin | 1 |
| qweqwe | 1 |
| fa | 1 |
| zsun1188 | 1 |
| 7ujMko0vizxv | 1 |
| raspberry | 1 |
| 1q2w3e4r5t | 1 |
| To: <sip:nm2@nm2> | 1 |
| service | 1 |
| oelinux123 | 1 |
| a1sev5y7c39k | 1 |
| oelinux1234 | 1 |
| fucker | 1 |
| CSeq: 42 OPTIONS | 1 |
| smcadmin | 1 |
| usr | 1 |
| sonar | 1 |
| vertex25ektks123 | 1 |
| sonar123 | 1 |
| vizxv | 1 |
| steam | 1 |
| Host: XX.XX.XX.XX:2323 | 1 |
| supervisor | 1 |
| Password1 | 1 |
| superzxmn | 1 |
| www.usr.cn | 1 |
| | 1 |
| 1qaz2wsx!QAZ@WSX | 1 |
| aA123456! | 1 |
| Accept: application/sdp | 1 |
| moxa | 1 |
| zjw | 1 |
| tech | 1 |
| qweqwe123 | 1 |
| Aa123456* | 1 |
| aDm8H%MdA | 1 |
| Content-Length: 0 | 1 |
| zyad1234 | 1 |
| test123 | 1 |
| 1qazXSW@ | 1 |
| Test1234 | 1 |
| Grand Total | 705 |
+--------------------------------+-------------------+
According to ZeroFox, here were the top 20 compromised passwords from 2022:
+------+------------+
| Rank | Password |
+------+------------+
| 1 | 123456 |
| 2 | 12345678 |
| 3 | admin |
| 4 | a11111 |
| 5 | 123456789 |
| 6 | password |
| 7 | 1234 |
| 8 | 12345 |
| 9 | secret |
| 10 | 111111 |
| 11 | 123 |
| 12 | 1234567890 |
| 13 | Password |
| 14 | 1 |
| 15 | admin123 |
| 16 | 1234567 |
| 17 | 123123 |
| 18 | user |
| 19 | P@ssw0rd |
| 20 | root |
+------+------------+
And what’d you know, only number 4 and 9 are missing from our list. Just goes to show how pervasive weak password hygiene is. Interestingly, there is some overlap between the usernames and the passwords. Many of the passwords look like the names of services we saw in our username list, sometimes with a few numbers added to the end. This brings back memories of the “solarwinds123” password for the notorious SolarWinds breach. I imagine the password list hitting my honeypot where geared toward service accounts with weak passwords that aren’t changed very often. I decided to check out of a few of the more interesting passwords to see if I could find out where they came from:
- 7ujMko0admin: The passwords appears to be the default set for the root account on some Unix systems. It’s associated with CVE-1990-0502.
- xc3511: This appears to be a similar case. It also seems it was used by the Mozi botnet to brute force Telnet.
- nE7jA%5m: This appears to be default creds for FiberHome routers.
Most of the other passwords I looked at appeared to be default account passwords, used in Telnet brute forcing, or seen in botnets attacking IoT devices.
Conclusion
Overall, I thought this was a very interesting thought exercise, and a good first step in my foray into threat research. While most of the traffic seemed to be very characteristic of the generic mass scans scouring the internet for low hanging fruit to exploit, it was still insightful to see first hand what those scans were doing and how they were doing it. Some of the IPs turned out to be somewhat interesting once I found the right thread pull on. My next step will be taking a look at the traffic picked up by the web server honeypot, and potentially even some of the firewall logs as well.
