Cloudflare Bootcamp

Twitter GitHub

Published: 2020-10-19

Filed under:

    Spectrum Bootcamp

    Problem

    • We couldn't protect from DDOS attacks services like SSH, RDP, Minecraft, etc
    • Cloudflare originally just proxy HTTP and HTTPS
    • Customers started to demand more services as Cloudflare gains market share.

    Pre-Specrum

    • Customers demanded a provider where they can protect all their workflows in the same place

    Solution

    Spectrum is a Cloudflare service that provides a DDoS mitigation service for any TCP/UDP traffic at L4 layer.

    Specrum

    Spectrum offers mainly DDOS mitigation services but also due to the size of our network, peering and Argo, Cloudflare can accelerate traffic and solve some problems related to latency and reliability on some applications

    Use-Cases

    Spectrum can work mainly in two modes:

    • Providing DDOS protection to non-HTTP traffic
      • Such SSH, Minecraft, RDP, RD Gateway, VPN-SSL, ...
    • Upgrading to L7 HTTP traffic in non-Standard ports.
      • Webserver running on ports different than 80 and 443. Like 8443, 8080,...
    Wait, What!!!!!????🤔🤔🤔

    Yes, after a while some prospects/customers ask for proxy at L7 some HTTP services not running on HTTP standard ports.

    menu

    So now you can effectively upgrade a Spectrum connection at L4 to L7.🤯🤯

    upgrade

    Getting the benefit from CDN, WAF, Firewall, or any other L7 services.

    ❗️❗️❗️Cloudflare can only upgrade and inspect to L7 HTTP/S traffic❗️❗️❗️

    The most common use-cases are:

    • Proxying SSH, Minecraft, VPN and RDP
    • Proxying HTTP/S traffic on non-standard ports at L7 (8443, 8080, 1234)
    • Proxying HTTP/S traffic on standard ports at L4

    Users / Customers

    Based on the common use-cases we can split the customer into 3 main groups.

    • Customers/ Prospects looking for a DDoS mitigation service for non-HTTP traffic. We can split this category into two,

      • Customer already using our web proxy service that has other workflows that like to consolidate and protect from DDOS. Ex: "I have my website on CF but I have this FTP service that I'd like to protect"

      • Prospect that her core business provides any type of non-HTTP services, this can be VoIP, WebRTC, VPN, Video,... some examples are Discord and Arlo

    • Customers/Prospects in the insurance, healthcare or financial sector that due compliance or internal regulations needs to protect from DDOS attacks, but their traffic cannot be inspected by a third party.

      • Those users will proxy HTTP/S traffic at L4 on Spectrum using TCP application rather than HTTPS application, so Cloudflare won't inspect any traffic.

    • Customers/Prospects with webservers running on non-standard ports. By far the less common case, but good to know we can ingest any Web traffic on ay port and put through L7 services if it is needed.

    Features

    • DDOS Protection

    • Basic Access List

      • Users can configure this access-list with IPs, IP range (/24 or /8), ASN or country acl

        ❗️❗️❗️This is the only ACL available, Spectrum doesn't allow to configure multiple ACLs

    • Argo Smart Routing for TCP traffic

      • Argo is available for all HTTP/S traffic, in addition to that, the Spectrum team has created Argo for TCP traffic, the main difference is that instead to use TTFB, use RTT to measure the best route to origin.

        Not available for UDP traffic, be sure that the customer's application runs over TCP before saying that we will accelerate with Argo.

    • Proxy protocol

      • Similar to X-forwarded-for or True-Client-IP, proxy protocol provide source/destination IP and port on the TCP packet, so the origin server gets the real information of who is connecting to the application. Two versions available
        • V1: Information is in plain text
        • v2: Information is on binary as a blob, weight much less than V1

    • SSL offloading

      • Always Off, always!!! 🙅‍♂️🙅‍♂️🙅‍♂️. If you think that you need SSL offloading on any TCP/UDP application, probably you are wrong. Please keep it OFF

    • BYOIP (Bring your own IP)

    • Load Balancing.

    What is coming?

    3 main features are the most requested by customers:

    • Static IPs
      • This is being actively worked on, probably for the first half of 2022 will be delivered.
    • Better Firewall capabilities
      • Ideally, Spectrum will be integrated with Magic Firewall so the customer can create more up to date rules to protect their Spectrum apps, product knows that but there are other priorities
    • Better analytics. Already shipped 🎉🎉🎉, Spectrum customers can use Network analytics, the ones are used for Magic Transit customers.

    Competitors

    There aren't many competitors:

    • Akamai
      • IP traffic accelerator
    • Fastly
      • N/A
    • Cloud Provider
      • Any with NGINX, but is not the same

    Qualifying questions for customers

    • Do you have any non-HTTP traffic?
    • Are you looking to protect that traffic?
    • What type of application is it?
    • What TCP/UDP ports use the application?
    • How critical is this application for you?
    • Do you require to know the True Client IP?
    • Do you need to load balance the traffic?
    • What type of load balancing do you need? RR, Active-Passive,...
    • Does your application need to be SSL offloaded? Why?
    • Does your application need to keep persistence? How do you do it at the moment?
    • Are you looking to improve the latency of your application? why? how much?
    • Does your application initiate calls/requests from origin to client?

    What protocols are supported?

    Spectrum is an L4 forwarding proxy so any TCP/UDP application is supported, during any customer call we should say that all the protocols are supported is they are TCP/UDP, but the experience tells us there are some caveats, all the protocols that require two channels to work will be problematic or tricky to implement, some examples are FTP, SIP, WebRTC, VPN. That being said we have already implemented successfully all of them on Spectrum.

    All the above protocols have two channels, control and data channel, usually, both channels run over different ports and the data channel is normally a random port selected by the origin.

    Let's see a real example, let's take VoIP.

    VoIP use mainly two protocols: SIP and RTP

    1 - SIP is the control channel normally run over 5060/UDP, when you call someone you send a SIP packet to the server, the server then finds where you want to call.

    2 - SIP server reply back with the information about how to start the actual call, the SIP response from the server will contain the IP and port client has to connect, "Connect to IP X.X.X.X on port 10011/UDP".

    sip

    3 - RTP is the data channel and normally uses a range between 10000 and 20000 UDP. As Cloudflare doesn't inspect the SIP traffic, we cannot know which port the origin server selected to the user connect to. Spectrum Strategy is open the whole range in Spectrum (10000 - 20000 UDP), so any port selected by the origin server will be accepted.

    4 - As you can imagine, the SIP response by default will contain the IP of the server, making that RTP traffic bypass CF, it won't be protected by Cloudflare.

    bypass

    5 - Customer needs to inject Spectrum IP on the SIP response to make the whole VoIP flow symmetric.

    correct

    I hope the above makes a bit more clear how those protocols work

    Supported protocols and Spectrum configuration needed

    • VoIP: ✅
      • SIP: 5060/UDP, 5060/TCP, 5061/UDP, 5061/TCP
      • RTP: 10000 - 20000 UDP

        Ask the customer which RTP range is he using

    • VPN: ✅
      • Clients VPN: All of them are supported (Cisco, Citrix, F5, Vmware,..), most of them are working over 443/TCP
        • 443/TCP

    Find the technical documentation for the vendor and you will know which ports are used

    • Site to Site VPN: IKE and IPsec are not TCP/UDP traffic but for compatibility with all the firewalls is encapsulated over UDP traffic, so we can proxy it, the main caveat is that VPN can be initiated just from one side of the tunnel. On the following diagram, you can see that S2S has to be initiated from the client-side and never from the origin side because you cannot connect to egress IPs. VPN

    Interesting Information

    • You cannot have same entries on DNS and Spectrum, it's one or the other

    • Technically same IP is used throughout the life of the app

    • DNS management

      • We talk with IAPI to add or remove DNS records on behalf of our customers
      • We create A/AAAA records on our own internal zone
      • We create CNAME records on customer zones to point at A/AAAA records on our zone

    • No Support in China

    • Regional Services supported

    • Logpush only

    • Network Ranges

      • 172.65.0.0/18 for FREE
      • 172.65.64.0/18 for PRO
      • 172.65.128.0/18 for BIZ
      • 172.65.192.0/18 for ENT
      • 2606:4700:0060:: /44 ANYCAST Proxy Anything Free
      • 2606:4700:0070:: /44 ANYCAST Proxy Anything Pro
      • 2606:4700:0080:: /44 ANYCAST Proxy Anything Biz
      • 2606:4700:0090:: /44 ANYCAST Proxy Anything Ent

    Troubleshoot

    1- My customer is getting this error when create a new Spectrum app.

    Banner

    • For each Spectrum application a IPv4 IP is assigned, to prevent customers waste hundreds or thousands of IPs, the default limit is 10, a customer can create up to 10 diferent hostnames on the Spectrum app. Spectrum assign one IP per hostname. Example: 
      - abc.pablo.com:80
- abc.pablo.com:443
- abc.pablo.com:22
- abc.pablo.com:4489
- abc.pablo.com:1234

    All the above consume just 1 IPv4 because the hostname is the same (abc.pablo.com) but:

    - abc.pablo.com:80
- ac.pablo.com:80
- ab.pablo.com:80
- bc.pablo.com:80

    The above will use 4 IPv4 because there are 4 diferent hostnames.

    • Most used Clickhouse queries

    Total Bandwidth by Application

    SELECT
    applicationTag,
    divide(sum(originBytes), 1000 * 1000 * 1000 * 1000)  as totalResponseTB
FROM proxy_anything_1m
WHERE date <= today()
    AND date >= today() - 90
    AND zoneId = 82376047
GROUP BY applicationTag
ORDER BY totalResponseTB DESC
LIMIT 500

    Determine Error Types by IP Address and Colo

    SELECT date,
       IPv4NumToString(clientIPv4) as clientIP,
       status,
       count() as occurrences,
       coloId
FROM proxy_anything
       WHERE date <= '2019-02-16' AND
       clientIPv4 = 1554702432 AND
      (zoneId = 4153684 OR zoneId = 111437752)
GROUP BY date,
          status,
          clientIPv4,
          coloId
ORDER BY date,
          status

    Spectrum + Argo: know if the connection is being accelerated and ingress/egress colo

    SELECT datetime, applicationTag, originTypeTag,  edgeTcpRtt, event,
substring(dictGetString('colo', 'airport', toUInt64(coloId)),1,3) AS colo ,
substring(dictGetString('colo', 'airport', toUInt64(originColoId)),1,3) AS exitColo
from default.proxy_anything
where zoneId = 30295432
and date < today()
ORDER BY datetime DESC
limit 100

    Top ASN by Application

    select clientAsn,
       applicationTag,
       dictGetString('asn', 'description', toUInt64(clientAsn)) as ClientASNDesc,
       count() as c
  from proxy_anything
 where date = '2020-09-06'
   and datetime between toDateTime('2020-09-06 12:42:00') and toDateTime('2020-09-06 12:48:00')
   and zoneId = ZONE_ID
 group by clientAsn,
          applicationTag
 order by c desc
 limit 10

    Number of Concurrent Connections

    SELECT
    toMonday(ts) as week_start_date
    , formatDateTime(toStartOfDay(ts),'%F') as date
    , zoneId as zone_id
    , round(avg(c)) as connections_average
    , max(c) as connections_maximum
    , quantile(0.99)(c) as connections_p99
   
  FROM (
        SELECT toStartOfMinute(beginTimestamp) as ts,
               zoneId,
               sum(activeConnections) as c
          FROM proxy_anything_aggregate
         WHERE date BETWEEN '2021-01-01' and '2021-03-31'
           and zoneId  = 191746237
         group by ts,
                  zoneId
         order by ts desc,
               c desc
       )
 GROUP BY
    week_start_date
    , date
    , zone_id
     
 ORDER By
    date asc
    , zone_id asc