Help A Newbie – Click Here For Our Famous ISP Survey!! DAwn McGatney's HOW ISPs REALLY WORK AND HOW TO START ONE MEAT FOR THE INTELLIGENT NETIZEN! This Web site contains no artificial ingredients… ALL of our pages are 100% pure old-fashioned hand-crafted HTML.
Page Added 31 October 1998 Last Updated 5 September 2000
Hi-
I'm DAwn's dog Wolf… A gift to Dawn from some jilleroo I was… way, way back… that year she spent the whole summer out on some sheep station, back when I was just some magical pup chasing higher-dimensional roos.
SO… Why am I writing these pages on how ISPs work and how to start one up? And on DAwn's Web site, which she's dedicated to helping folks onto a decent ISP?
First, because DAwn asked me to… And sometimes, it's VERY difficult to say “NO” to this woman.
Second, because it seems to me that the better folks understand just how an ISP really works, the better they can make their decision on selecting one. (You will definitely understand why Usenet drops articles, or why not all ISPs are 56K, or why reading a Traceroute is useful, etc.)
But third, and foremost, because DAwn gets so bloody much mail and feedback from folks asking her about starting an ISP and inquiring about their innards.
(And to be honest, I was myself curious to see exactly what was inside that Magical Mystery Tour named an ISP, the thing that my PC dials up and who connects me to the magic of Cyberspace… this thing that folks love and love to hate… and so this is a tour we'll both be taking together.)
I'll try not to duplicate too much stuff that appears in my classic work How The Net Really Works. And whenever you encounter some term you don't understand, just look it up in Dog Wolf's Glossary.
At the outset (just before Halloween of 1998… wooooooooooo… these colours should give away the fact that I'm anticipating Satie night's party… I'm coming dressed as a Cisco router), let me state that, like some fine wine, my dog Wolf pages improve with age (and with Feedback from knowledgeable gurus out there).
And at the outset, let me also state for the record… Anyone who tells you that an ISP is a money tree that any Unix-illiterate lamer can partake of in some cake walk is, IMNSHO, misleading you.
Sorry… wish I could say that setting up an ISP is a breeze, and that anyone can do it over a spare holi weekend… but that just ain't the way it is. This page is NOT intended to be a guide for folks setting up a commercial ISP.
So, this having been said, let our adventure begin. And as always, I refuse to talk down to folks… these pages are NOT “ISPs For Dummies.” They're ISPs for intelligent newbies. Cool.
And Happy Halloween.
DAwn's Pumpkin Patch
A SIMPLE ISP
It would be cool if we could just add some program to our PC that would let our modem dial up the Net directly, for free, without any ISPs. But that isn't how it works.
Our modems connect to our telephone lines… the switched public voice phone lines… phone lines which are both analog and switched. We want to hear the weather forecast here, we dial 410-936-1212.
We want www.weather.com… IP address 208.134.241.155 on the Net, we just dial with our modems… bzzzt, thanks for playing… because our modems can't dial an IP address. They can only dial telephone numbers. And the Net is NOT on the switched voice analog network, the public phone network, that we all know and love. The Internet lives on its own “semi-private” network.
An ISP is that something that allows folks with PCs and modems who have a connection to the voice telephone network (or, ok, to some cable TV system – same thing really) to connect to the Internet… to access the Internet… to actually become a part of the Internet.
An ISP interfaces for us between the public phone system (or some cable system) and the Internet's semi-private digital phone lines… phone lines that carry PACKETS instead of voice conversations.
REALITY CHECK… IS THAT ALL THERE IS?
Reality Check Subpart A...
An ISP allows us to access the Net. Cool. Ok, intensely cool. But that's not really ALL that an ISP does for its customers.
ISPs today usually also retrieve and store subscribers' e-mail. They access the Web and host customer Web sites and cache Web pages. They provide varying amounts of “content.”
ISPs save and provide access to zillions of Usenet news articles. They provide us with disk space for things like FTP. They let us chat in chat rooms.
And so on. (Just wanted to clarify this.) If all an ISP does is provide connectivity from phone lines to the Net (and out back again), it had better do it extremely fast and well; most ISPs today provide at least SOME of the value added services we've mentioned. So there.
Hold this thought in the back of your mind; we'll return to all the “extra” ISP functions gradually… like going out into the ocean in mid-June when things are still intensely cool.
Reality Check Subpart B...
We received on this awfully hot and humid July 1999 day in Hunt Valley the following question…
“Is there perchance a way a fellow could hook up with a backbone? I know this sounds like I'm spineless, but I was referring to an Internet backbone provider; I do not want to be an ISP or anything like that, I just want to hook up fast. So fast that my ears ring. I read through and through all of your wonderful, informative, witty and entertaining info, did I miss something? Can this be done?”
And your dog Wolf responds… Absolutely. (And no one who can review our article with the accuracy and precision you exhibit could ever be “spineless.”)
You may not be able to get dial-up service to your favorite backbone (like UUNet – but then again, you may). But backbones would love to sell you, the non-ISP, a piece of their bandwidth through a T-1 – or even a slice of a T-1 (called a “fractional” T-1) – like 384 k-bits/ sec for example.
In fact, if you have more than one PC on a Local Area Network (LAN) in your home(?) or business, that's the way many folks connect to the Net… run a T-1 from your LAN to ATT or PSINet or whomever. Just don't forget the router and the CSU/DSU(s).
For an initital payment of a few thousand dollars, plus a few thousand each month, you can surely connect directly from your home to a backbone over a T-1 and never see an ISP. Except that YOU will become the de facto ISP. Because you will be handling your own Usenet, E-mail, Web hosting, DNS, etc.
Good luck.
REVIEW OF PACKETS
Right-o. Time now for a brief but very important review. (See my Dog Wolf Classic The Net for all the details.) On the Net, we send and receive things in packets.
If we want to send an e-mail, TCP/IP breaks our e-mail into tiny packets. And TCP/IP writes the destination address on each packet. And as a result, TCP/IP can send each packet separately over the Net... like millions of cars on the Interstates, each holding up some big sign saying, "Key West or Bust."
The Net is a lonely place. Each packet travels the Net alone. The many packets from our one e-mail may each take a different route across the Net. In going from Maryland to Santa Cruz, California, one packet of my e-mail may go through Dallas, while another may pass through Chicago.
Our packets may even arrive at their destination at different times and out of sequence... but the receiving computer is able to reconstruct our original e-mail.
Routers (intelligent hardware devices, our friendly traffic cops along the Interstates) read each packet's destination and forward each of them along the best route available at that moment. And packets are small... usually only a few hundred bytes in size.
Why? Well... you can ship all of your watermelons on a one mile-long freight train. Or you can load the melons onto hundreds of trucks. And if one truck breaks down, it's a lot easier to fix or reload than cleaning up after our train runs off the tracks.
If there's an error in transmission, it's a lot easier and faster just to resend one small packet. PLUS... small packets tie up routers only for short periods of time. They're in and out fast.
And that's all we need to say for now on packets.
Now let's assume that we're an ISP, and we have a customer who wants to send something from her PC across the Net to her friend… an e-mail. (Folks send lots of e-mails.)
Her Winsock Program happily running on her PC (it better be, if she wants to play “Internet”) chops her e-mail into Net-sized packets. And her PC then shoves the little packets into her modem, which converts them into squeaks and squawks. That's so that the packets can travel across an ordiany voice telephone line, from her PC to our ISP.
So to begin with, our ISP needs a modem on each incoming phone line, a device to convert our customers' squeaks and squawks back into Net packets. Because we can't put squeaks and squawks (or voice, or fax) onto the Internet.
Packets… that's all the Net knows. Maybe our ISP is very small, so that we have only one incoming phone line going into one modem… or maybe we have a few incoming phone lines and modems… whatever. Our ISP's modem(s) will convert the analog squeaks and squawks back into digital Net packets. Cool.
HOW MANY LINES
How many phone lines does our ISP really need? Well, since we're just starting up our ISP, it seems that we should be able to have just a few. Maybe we'll let our friends dial in for free while we learn about running an ISP. Maybe we'll all share the cost.
But as out ISP grows, and we get more and more customers, one phone line for every six or seven customers is reasonable. As our ISP grows even bigger, we might go to 10 customers per line. If you'd like a little queueing theory with catsup, you can peruse DAwn's intensely cool page on ISP User-To-Modem Ratios.)
Otherwise, look at it this way... the bigger our ISP is, the more customers we can have per modem. Because the more customers and the more phone lines and the more modems our ISP has, the more likely it is that some customer will free up a modem just before some other customer is ready to connect.
To a LARGE extent, how many lines we need depends on how we price our ISP's service and what type of customers we attract. (And we always need to bear in mind that it may take a couple of months to get additional new incoming phone lines and modems installed.)
SO… at this point, if we can just shove the packets bursting forth from our ISP's modems out to some bigger ISP, out to some ISP who is connected to the Internet and is willing to resell that access to us… well, by golly, I think we'd be off to an intensely great start.
We'll probably want to run the packets through some PC at our ISP on their way out the door. That way, we can handle mundane stuff – like letting folks sign on.
The simplest ISP might work like this… we buy some cheap (used?) computer, and we install free Linux (a flavor of Unix) on it. We buy a dedicated connection to some bigger ISP; and we buy, say, four dial-up phone lines from the telco, and four modems.
Three of our phone lines and modems we'll use for incoming calls from our customers. And the fourth phone line and modem, we'll use to dial-up OUR ISP. (And since it's a dedicated connection, we dial our ISP once and leave it connected.)
Believe it or not, this will actually work in the real world. We have just put together an ISP.
UNDER THE COVERS
Careful... a great many ISPs today support 56K... How do they do this? What does this mean for our ISP?
ANSWER... We "exagerated." Recall when we said that our customer's PC breaks e-mail (e-mail is digital... it's bits... 0's and 1's... the only thing that our simple-minded PCs can comprehend) into packets (also digital) -- which feed into her modem and come out as squeaks and squawks onto the phone line (analog... the voltages look just like a cross-sectional slice of terrain... hills and valleys of almost any height... not just "0" or "1")... and the squeaks and squawks travel over the switched public voice phone system and come into our ISP as squeaks and squawks? Recall that we said this?
And recall, at the very start, where we said the Internet was digital, and the public switched telephone network (PSTN) was analog?
Fantasy. All (well, almost all) fantasy. Sorry.
Time for some reality. It's nearly the Year 2000, times have changed, technology has marched on, etc. These are exciting times for networks of any kind.
Today, when the squeaks and squawks from our customer's modem reach the phone company's central office (usually just a few miles from her home), the phone company converts the squeaks and squawks immediately BACK to digital format... back to 0's and 1's... before putting the data onto the rest of the PSTN. And it travels to our ISP as such.
In other words, stuff travels over most of the public switched telephone network in DIGITAL format. (It's just easier and cheaper and provides better quality service when the phone companies do it that way.)
"So what?" -- You ask your dog Wolf. Answer... we have an OPPORTUNITY here.
The old(er) V.34 modems are still GREAT modems, but they were designed on the assumption that data coming from our ISP would be converted from digital packets to analog squeaks and squawks AT OUR ISP, and then from analog to digital (A-D conversion) at our ISP's local phone company office.
And this is the bugger... converting from analog to digital at our ISP's phone office is the Kiss of DooDoo for transmitting data faster than 33.6 kbps.
WHY? Because this A-D Conversion introduces noise (impress your date... it's called "quantization noise"), and it is that noise which limits v.34 modems to 33,600 bits per second.
BUT IF... if... if the packets coming FROM our ISP to the phone lines could remain as digital, all the way to our customer's phone company office, and if they were never A-D converted... and if our customer is not too far from her phone office... then yes, we can send data at 56,000 bits per second (or thereabouts) from our ISP to our customer.
(The stuff our customer uploads to our ISP is converted from analog to digital at her phone office... and recall that A-D conversion is the Kiss of DooDoo to data... so our customer will still be sending stuff to our ISP at a maximum 33,600 bits per second, due to quantization noise from this A-D conversion. BUT, if we do things just right, our customers can download data from our ISP at "56K" . . . which is Ok, because most folks do a lot more downloading from an ISP than uploading TO an ISP.
(If our customer is wise and foxy like some collies I know, then she may be able to get an ISDN line to her phone company... so that her data will remain in digital format from her to us the whole way... so she can receive AND SEND at 56K.) )
Quick review--> D-A (D to A) is OK, but A-D is sheep dip (sorry).
So cool... but what does all this mean to our ISP? It means $$$ if we want to offer our customers 56K service. It means that to support 56K, our ISP must have a digital connection to the PSTN. That is, to provide 56K service, our ISP must connect to the phone system with a digital circuit; like an ISDN line, for example.
We have to avoid an A-D conversion.
The ISDN digital line (or perhaps the faster, more pricey digital "trunk-side T-1") coming from our phone company will then need to connect to our Digital Modem, like a 3Com US Robotics MP I-modem, or a NETServer I-modem, or a Courier I-modem, or a SuperStack II Remote Access System 1500, or a Total Control remote access concentrator... etc.
56K digital modems are integrated into these pricey "access servers," which combine a modem and a terminal server (we'll explain that shortly) into a single, integrated, expensive box.
Many access servers (like Sun Microsystems cool boxes) handle up to 48 dial-up connections. SO... if our ISP has 2,400 customers -- and if about 10% are dialed in at once -- then we'll need 5 access servers... 48 x 5 = 240.
NOW... watch my hands, class (closely). Notice that stuff coming FROM an ISP can download at 56K, right (if all is well)... but that stuff going TO an ISP can't go at 56K... because in going from our customer's PC to our ISP, the bloody Telco performs the dreaded Analog-to-Digital (A-D) conversion. (Remember, DooDoo?) Ok... This has one additional ramification for us.
Assume that we are a very small ISP, and that we decide to connect to some bigger ISP who is connected to the Net. And assume further that we want to do Web hosting. Well... when folks in Lahaina on Maui (just daydreaming) request some Web page from us... some page that we're hosting... we have to send it to our BIGGER ISP to get it onto the Net. Right? Right.
Can our ISP send it at 56K through a 56K modem? Can our ISP send to another ISP at 56K?
NO. Noise. DooDoo. And so, if we're connected to our bigger ISP just by a dial-up modem (as we propsed above), we can only transfer our pages up to the Net at 33.6 kbps. Is it a good idea to do Web hosting when you're connected to another ISP by a dial-up modem? Answer... NO.
(Some folks actually attempt to do this without buying a dedicated connection, thinking all it takes is the famous "static IP" address. However, most subscribe to a dedicated connection on the Bigger ISP.
But some don't, and just stay connected forever... which is one reason why, your dog Wolf believes, we're seeing more and more ISPs drop flat-rate dial-up pricing.)
And that's the why and how of 56K.
REALITY CHECK… TOO FAR FROM THE PHONE OFFICE?
When we started talking about 56K, we said something like, “If our customer is not too far from the phone office, it will work.” We usually hear 3 miles or less for 56K to work really fast.
DAwn and I live north of Hunt Valley, Maryland, out here in the sticks. We're about 20 miles from our local phone office. And like many folks out in the sticks, we never connect below 49,666… usually at 50,666. (No, the 666 is not an omen; it signifies bit-robbing by Bell-Atlantic.)
HOW? Because Bell Atlantic takes our phone lines and others in the area and converts them all to DIGITAL way out here – near to us. And then everybody's stuff travels the last 18 or 19 miles to the phone company building on some T-1 (or faster) digital fiber optic cable.
Live out in the sticks? TRY a 56K modem. You may be just as surprised as DAwn and I were. Remember, Year 2000 is just days away. (And if it weren't for days, there'd be no Year 2000.)
CABLE MODEMs — DARE WE ATTEMPT?
Well, dog Wolf has now explained to you the ins and outs of 56K… but dare we to fathom that modern phenom, la Cable Modem?
Sure, what the hay… ain't nothin' here to be 'fraid of. (Just don't try this one at home… the marriage of cable video technology with Internet technology is for the beeeg boys (for now)… AND… understand that not ALL cable modem systems work the same way or provide exactly the same speeds.)
Ok… Recall how our old-fashioned standard modems takes packets from our ISP and converts them into squeaks and squawks… so that they can travel along our analog phone lines, swim downstream to our phone company building a few miles away and ultimately to some customer of ours?
Cool. Well, at our Cable ISP, TCP/IP packets are converted, not into squeaks and squawks, but into blues and reds and video… just like the latest Die Hard XXI movie, whizzing merrily along some cable to our TVs.
In a Cable ISP, our TCP/IP packets are converted into analog video signals… pictures, if you'll allow.
Why, you ask your dog Wolf, why? Well, because, mate, that's what the cable system was built to handle… blues and reds and Die Hards XXI.
The packets from our Cable ISP, in video format, travel downstream along the cable, using some unused TV channel, say channel 87.
And all these “video packets” on channel 87 don't interfere at all with C-SPAN's live broadcast of the Australian Parliament on channel 86 or with the Seinfeld reruns on channel 88.
And stuff can move along our channel 87 at about 36,000,000 bits per second (not too shabby).
The same cable that carries your customer's TV fare also carries her Internet packets… Cable is a manly, robust medium, with a BEEEG bandwith (assuming that your cable company has upgraded to fiber).
Each customer has a “splitter” in her home… one piece of cable runs from the splitter to her TV(s), and other piece of cable goes to her PC(s).
AND… when the video signal reaches our customer's PC, a CABLE modem sitting nearby converts the video back into packets… voila.
And stuff can now move downstream into our customer's PC at about 5,000,000 bits per second.
(Yea, cable can transmit data at 36,000,000 bits per second to her cable modem, BUT… that Ethernet interface card connecting her cable modem to her PC can only pass data into today's PCs at 5,000,000 bits per second at best (still not terribly shabby). )
Well… not quite. At our Cable ISP, our server, called the “Head End” (Aussie term) Controller caches popular Web pages. But for FTP stuff and less popular Web pages, we're still out on the slooow Net… and so, our customers are NOT going to download at 5,000,000 bits per second from the Net – not in general. Fast yes, but not this fast.
(Like always, our Cable ISP connects to the Net (or to some bigger ISP) through a router via a leased line (like a T-1 or a T-3). Our router also connects to our Cable ISP's Head-End Controller (usually using 100 Megabit Ethernet). )
CUSTOMER'S—–ETHERNET——-CABLE
PC CARD MODEM
|
|
|
ROUTER———FAST——–HEAD-END
| ETHERNET CONTROLLER | |
T-1 or T-3~~~~~~~BIGGER ISP OR THE NET
What about packets moving UPSTREAM?… packets moving from our customer's PC up to our Cable ISP?
Well, they go through the Cable Modem sitting on top of our customer's PC and get converted into video just the same way.
The only difference is that upstream stuff going TO our Cable ISP runs through the cable system using some very LOW channel. Say channel “-10” (yea, minus 10).
And this is a channel that's subject to lots of interference. (Hey, there was never any TV here on channel -10, so all the CBs and garage door openers and model airplane controllers in the world were assigned frequencies here.)
(Some cable ISPs, such as @Home, now limit the maximum rate at which a modem can send data upstream to 128 kbps – though @Home doesn't like to discuss this. The purpose of the upstream data rate cap, according to an @Home rep, is to prevent what he called “subscriber abuse,” particularly the running of a Web server from an @Home account. Whatever.)
SO… partly because of all this Schmutz (another Aussie word) along the upstream link from our customer's PC to our Cable ISP, the link from the customer to our ISP will run at perhaps 500,000 to 1,000,000 bits per second.
Impress your date… Stuff going upstream from your customer's PC is sent on the noisy 5-40 MHz band. Where exactly? Dunno. (Trust me, DAwn dates men who are impressed by this.)
A cable modem has to be able to send stuff on ANY 2 MHz slot in that 5-40 MHz range… like 5-6 MHz, or 7-8 MHz… her modem, and everyone else's modem on the system.
NOW… if our cable system uses glass fiber instead of coax wires to send stuff (we hope), we're a LOT less susceptible to noise when swimming upstream (like a salmon – great source of omega-3).
But fiber upgrades cost mucho dollars, which is why smaller cable systems often throw up their hands and make you send data to them the old way… via your telephone lines.
If a cable system has fiber, the signals sent downstream from our Cable ISP are converted by laser transmitters to optical signals; then just outside each group of houses, a laser receiver converts the light pulses back into electrical signals, so that they can travel along the coax into each customer's home.
Anyway… how do 1,000 customers send data on the few slots available between 5-40 MHz. How do we do that, dog Wolf?
Remember something called a “Head-End Controller?” The Head-End Controller tells our customer's cable modem WHEN to send stuff and WHICH 2MHz SLOT to use at the moment.
And yea, about 1,000 customers normally share this fairly small bandwidth… But hey, cable systems were designed to DOWNLOAD stuff to your TV, not to UPLOAD stuff from a customer's PCs.
The Head-End Controller is like some air traffic controller… it tells the planes when they can take off and which runway to use.
And this is why (for now), the cable company, who owns the head-end controller, also gives you ITS choice of cable modem… no standards are here yet… you have as much choice in selecting a cable modem as you have in deciding to whom to pay your federal income taxes.
(Really impress your S.O…. How does the head-end controller tell a cable modem which 2 MHz slot to use and when to transmit? (Now you know why I chew up DAwn's Italian designer shoes… I've made a 1999 New Year's resolution to stop… we'll see.)
ANSWER… one 2 MHz slot is used for control information. When our customer powers on her cable modem, the modem first searches all the available 2 MHz slots. It finds the special “Air Traffic Controller Information Slot” (yet another Aussie name), and it listens there for instructions.)
Now recall, in your customer's cable modem, the packets coming from our Cable ISP in video format are NOT converted directly into TCP/IP packets.
Instead, they're converted first to ETHERNET packets, and then fed to an “Ethernet Card” in her PC, where they're finally converted back to TCP/IP packets. That is, the cable modem communicates to and from our customer's PC through an Ethernet interface card. WHY?
Well… our customer can't connect the packets coming out of (or going into) a cable modem to ye olde serial port, like COM2; serial ports have a maximum speed of 115,200 bits per second, and cable is MUCHO faster… ¿Está claro?
And that's the Cable ISP story. For now.
What does our simple ISP cost us?
We'll pay $80/mo for 4 ordinary phone lines, $150 for four used 28.8 modems, say $100/mo today for a dedicated PPP line to our bigger ISP at 28.8 or 56K, $500 for some used PC (a 486-66 may be fine), $20 for a copy of Linux on a CD and the cool Red Hat book that explains how to configure the beast.
So, for about $700 in one-time costs, plus $180/mo, plus some one-time setup fees to the bigger ISP and to the phone company for our phone lines… voila… we have a real, working ISP.
If this approach seems simple-minded, it is. But all an ISP basically does is to move IP packets to and from some National Backbone provider – some provider who can reach any IP address on the Net. Our ISP now looks like this…
PHONE LINES~~~~~MODEMS——-LINUX PC To Our ISP At Our ISP At Our ISP
|
|
|
Bigger ISP~~~~~PHONE LINE~~~~~MODEM
| To Bigger ISP At Our ISP |
The Net
ENTER THE T-1
Time passes, we think we're going to get a surge of customers soon… what should we do, besides adding more incoming phone lines and modems?
Answer… our 28.8 dial-up connection to the bigger ISP has got to go. So how can we REALLY connect our ISP to some bigger ISP who's willing to let us resell its connection to the Net?
Hmmm… well, we could always lease a T-1 dedicated phone line. That'll let us connect to the Big ISP at 1,544,000 bits per second… not bad at all. It's about 30 times faster than a 56K dedicated line.
(A T-3 runs at 45,000,000 bits per second and costs about $10,000 per month… if your ISP is doing some really heavy Web service for folks, you may want to start thinking about one… but NOT to connect to some PC running Linux.)
We can probably convince a big ISP to sell us a T-1 connection for about $1,000 a month (maybe even less). And the actual physical T-1 that we lease from the telephone company will be another few hundred dollars each month. But if we can sign a long-term lease for the T-1, we'll probably get a break on the pricing.
What's a T-1? A T-1 is just two pairs of copper wires connecting two points; one pair is used to send packets, the other pair is used to receive packets. (A T-3, on the other hand, is usually fiber-optic, not copper; it can move data at about 45,000,000 bits/sec… a big pipe it surely is.)
Do we really need a T-1 to connect to the bigger (“upstream”) ISP? No, technically not. We can improve our 28.8 dialup connection with to the upstream ISP with a dedicated 56K line, but bear in mind that things like hosting Web pages can cause a lot of data to move upstream from our little ISP. If we're hosting Web pages for our customers, we'd better think T-1 (or even faster).
UNDER THE COVERS
Careful... sometimes it only LOOKS like we have a dedicated leased T-1 line. In reality, most long distance carriers use switched circuits to provide what appears to be a dedicated T-1 line. (These simulations are called "Virtual Private Networks" or VPNs.) But our T-1 WILL probably be some real, dedicated copper wires at least from our ISP to the phone company's nearest central office.
Next point... Real ISPs usually prefer TWO T-1s. For one thing, that gives them a bandwidth to the Net of over 3 Mbps. AND... if the two T-1s attach to two DIFFERENT Bigger ISPs, we reduce the chances that a single network problem will cut off our entire access to the Cyberverse. It's called redundancy; and the bigger the ISP, the more of it you should find.
Final point... Our T-1 (or pair of T-1s) preferably connect to a hardware thing called a router. More on routers very soon.
With a T-1, our ISP can probably support a few hundred incoming lines and modems. And if we assume a 10 to 1 customer to modem ratio (yea, there'll be a few busies), we now can have perhaps 2,000 customers. And at $20/month per customer, we've already paid for our T-1.
(No, it's NOT quite that simple… like, heavy FTP and Web usage by our customers could drag way down the number of customers our single T-1 could support.)
THE NITTY GRITTY
If we grow until we're a "Bigger ISP" to some folks -- and if we then allow OUR customers to connect to our ISP with their T-1s -- there is a rule of thumb called "5 to 1." It means that we want to allow our customers to connect no more than five T-1s to us for every T-1 we've connected to the Net. And even this ratio only holds if our customers' T-1s incoming to us don't all get busy at the same time.
(What the pros do is monitor their T-1 (or whatever size) connection they have to the Bigger ISP (or directly to some National Backbone operator); and when that pipe to the Net begins exceeding 50% busy, then they start adding MORE capacity between their ISP and the Net.)
Where do we raise the capital for our phone lines and modems and T-1 and all the other stuff we haven't talked about yet? That's something we won't get into… but when you succeed, let your mate dog Wolf know.
Sure, we could charge our customers a small fortune; but today, most folks are savvy enough to know that high prices don't always mean that our ISP is investing in fast, high capacity connections to the Net.
And they know that even if our ISP IS investing their payments in high speed links, our ISP still may not be employing a decent network topology to provide redundancy and sufficient bandwidth. Whatever.
When last we spoke with our ISP, with our T-1, it now looked like this…
PHONE LINES—>MODEMS—>—>CPU—>T-1—>BIG ISP—»Net
ENTER THE MORE REALISTIC ISP
That's ISP-ing for pleasure. Now let's bring our ISP a little bit more into conformity with the real world… with the way folks actually do this thing called ISP-ing, for profit and pleasure.
ENTER THE CSU/DSU
Ever heard of a CSU/DSU? No? Ok, it stands for “Channel Service Unit/ Data Service Unit.” And it's a piece of hardware that we'll NEED when we connect to any high-speed digital line, like our ISP's T-1. (Think of it as a digital modem.) Our T-1 line is going to need to connect to “us” through a CSU/DSU.
Why dog Wolf, you ask… tell us WHY we need this CSU/DSU?
Grrr… Ok… just briefly. First, a CSU/DSU protects the T-1 (or other leased line) that belongs to the phone company.
And second, it converts our standard computer bits (0 and 1) into the kind of bits (bipolar) that the phone company's lines expect. All bits may be created equal, but some bits are more equal than others – at least that's what the phone company says.
We think of the CSU/DSU as a hardware thingy that cleans up the bits on a high-speed line so that they're fit for a router. Because a high-speed line from the phone company can't be connected directly to our router (whom we'll introduce in uno momento).
Our router doesn't speak “telco bits,” just “computer bits.” The Bigger ISPs, who provide us access to the Net, often require that we buy a CSU/DSU for our end of the leased line to them – AND for their end too… and they'll often require a specific brand. For a T-1, expect to pay $1,000 or more for each CSU/DSU
Just keep in mind… Because our T-1 is ALREADY digital, we don't have to convert our packets into squeaks and squawks… we don't need a REAL “outgoing” modem when we connect our ISP to a T-1 on its way to the Net Cloud.
(Net Cloud? In almost every Net diagram we've ever looked at, the Internet, for some reason, is represented by a CLOUD with “Internet” inside of it. We'll refrain – we don't have any cloud graphics.)
So, our ISP now looks (or will soon look) like this…
Server Server
\ /
Server–ETHERNET HUB–> ROUTER–> CSU/DSU—> T-1
/ \
Server Server
ENTER THE ROUTER
It's cute and cuddly that we started our ISP with our “little baby girl” Linux PC. But out there in reality, things are tough and often cruel, my mate, when trying to run an ISP with just a single CPU box. (This is one reason why Sun Microsystems has grown so large – it takes lots of “boxes” to build a manly, robust, muscular ISP.)
And so we add more processors. Perhaps we'll decide to add a mail server, and a news server, and a Web server.
And when we do… when we have two or more CPU boxes, well… we'll need some way to connect them… to network them together, so that they can exchange data. And so, now at our own ISP, we'll have created our own little network. And we'll need to connect that little network to the Big Internet. Carefully.
Why carefully? Because some packets belong just to us, to our tiny local ISP network… maybe we route messages from my work station to yours at our ISP, along our Ethernet, trough the walls of our offices… “Lunch time, let's send out for some Szechuan, extra hot and spicy” (as DAwn would say).
Do we really WANT packets like that leaking out onto the Internet? Yet on the other hand, we may well use the same local Ethernet to carry genuine Internet packets.
Ok, now… Ever heard of a router? Yea, those things that route packets out on the Internet… the traffic cops who stand along the Beltways of the Net (called NAPs) and wave our packets onto the correct Interstate (National Backbone) to reach their destination addresses.
Well, surprise… because now that our ISP has become a little network of its own, we're going to need a router for every digital leased line connected to our ISP.
So far, we just have one digital leased line – our shiny new T-1; so we'll need just one router to connect our ISP's little network with our T-1… one router to keep our internal “Szechuan packets” from leaking out onto the Net, while still letting our customers' e-mail packets move freely onto our T-1.
Routers are used to connect our little network at our ISP with the Internet at the far end of our T-1 phone line. Routers connect networks, and they filter and isolate traffic.
We need a router to determine which packets belong to us alone and should thus be isolated and remain in our little ISP's network (that is, should stay inside our ISP's building… our “Szechuan packets”), and which packets are destined to venture out onto the Net.
And of course, in the other direction, our router allows packets to come in from the Net. But our router – our traffic cop and security guard – permits only VALID packets to enter from the Net.
Routers actually read the destination addresses on our Net packets (“Key West or Bust”), and they don't allow packets with bad addresses to get out onto (or to come in from) the Net.
Yes, routers surely are smart; and they are kind of expensive… Routers were invented by some very smart folks (you can tell that when you go to program a router); and because routers can connect different “little networks” together, routers make the Net possible… and your dog Wolf thinks it's time we got some experience with those routers… and so, our ISP shall now include a manly, robust router… perhaps a Cisco Router, the most popular brand today.
In June 1999, Cisco introduced its new low-end 2600 series router for as low as $2,000 – with 64 Meg and a RISC-based CPU.
REALITY CHECK… OUR ISP'S FIRST LINE OF DEFENSE
Our ISP must control intrusion into its little network – and into each and every server on its little network.
We guard our ISP with routers and firewalls. The router – with its capability of filtering packets – is our first line of defense.
Our router can direct packets (or block them) based on their source IP address and their destination IP address. It also controls packets based on their PORT address – a number signifying what service a packet expects.
Thus, by checking for a PORT number of “80”, our ISP can make sure that ONLY HTTP requests reach our ISP's Web server.
Unfortunately, slick as they are, routers can't log their actions – and our ISP can't rely on routers alone to detect or trace intrustion attempts. In the real world, we add firewalls to our packet filtering routers.
We set up our router to block all inappropriate requests of any server, say our e-mail server; our firewall then performs a more detailed analysis of packets, dropping any that are suspect.
So now, our ISP looks like this…
PHONE LINES—>MODEMS—>—>CPU#1—>CPU#2—>
—>ROUTER—>CSU/DSU—>T-1—>BIGGER ISP—>Net
HOW DO REMOTE POPs WORK?
Right-o, we're really getting into the good stuff now.
At this point, I hear folks muttering to themselves about POPs – Points Of Presence – the numbers that you dial up to reach our ISP's modems. Sure, the diagram we've sketched above is a cool beginning – IF all of our modems are located in Mytown.
But we're all savvy Netizens here, and we KNOW that many ISPs have dial up numbers in more than one town.
And while our ISP is going to have some local modems in Mytown, we'd also like for it to have a phone number over in Yourtown.
That way, customers in both towns can dial up our ISP as a LOCAL CALL, no matter where we may actually locate the innards of our ISP. So how? How does this happen, dog Wolf?
(I doubt that DAwn gets ANY question so freqently as… How do ISPs do POPs?… except, of course, “Are you busy Saturday night?”)
AN ISP IN TWOs CITIES
Let's first look at adding another city to our ISP. Actually, adding modems in one or two additional cities can be done very easily. Voila… An ISP Of Two Cities…
PHONE LINES——> MODEMS——> ROUTER——→ CSU/DSU In Yourtown In Yourtown In Yourtown In Yourtown
|
|
(Yourtown) | ……………………………………………|… (Mytown) |
|
|
|-------<------- Leased Phone Line -------<------|
|
CSU/DSU
| |
ROUTER—> CSU/DSU—> T-1—> BIGGER ISP—> The Net
|
|
|<-----CPU#2
|
|<-----CPU#1 <-----Modems <-----Phone Lines
In Mytown In Mytown
(Yes, we've slyly connected all the pieces). Remember that the stuff above the dotted line is remote – way off in Yourtown. And the stuff below the dotted line is in our ISP's building – over here in Mytown.
How fast is the “Leased Phone Line” in our drawing above? Well, a 56k leased line from Mytown to Yourtown could support up to 12 remote modems.
Above that, we'll need a T-1 (or fractional T-1). But now, both you (in Yourtown) AND I (in Mytown) can dial a local phone number to connect to our ISP. Cool, yes? Yes.
We may decide to invest (mucho $$$) in a Total Control Modem Rack at our Yourtown site – which would allow us to control our modems remotely; then – should there be a problem in Yourtown – we won't need to drive (or fly) all the way to our remote POP to fix it.
Our remote POP can be in offices that we lease in Yourtown, or in some building that we have a “co-location” agreement with. (Co-location means we run our equipment in their building.) Or we may rent space from the telephone company in Yourtown and just put our stuff in there.
Or alternatively, we can just set up a phone number for our remote POP – and instead of actually installing modems and terminal servers, we can simply have the phone company route calls coming into the remote POP's number to our ISP's headquarters.
This has the advantage of eliminating maintainance of the remote equipment. It has the disadvantage that we'll have to pay a long distance charge on each call to our remote POP. Folks often call this arrangement where the phone company handles the whole thing a “virtual POP.”
Whichever approach we use… it all works. And this is how you add another town or two to your ISP's customer base. But what if we want ISPs all about the US… or all over the world? What then, dog Wolf, you ask?
HOW TO GET POPs IN HUNDREDS OF CITIES
Did you ever notice how, say, in Yourtown, USA, the folks who subscribe to DogWolf.Net and to Gerbil.Net and to Aardvark.Net... they ALL dial the same POP number? Hmmm...
In reality, the POP number belongs not to any of these ISPs, but instead to UUNet or PSINet or MegaPOP.
Companies like MegaPOP are called "Bulk-dial providers" or "Wholesale Dial-ups," because they offer modem connections for OUR ISP's customers at wholesale rates.
And ISPs who use MegaPOP (or PSINet, etc) to give them POPs in other towns are called MegaPOP (or PSINet, etc) resellers. Because these ISPs are in effect "reselling" MegaPOP's dial-up access.
Like, in Minneapolis, UUNet has a large pool of modems that's shared between GTE customers and Earthlink customers and MSN customers and other ISP's customers.
UUNet is providing wholesale dialup services for these ISPs. AND... since UUNet has one bank of modems in Minneapolis, everyone in that town who subscribes to one of these ISPs dials the same number to get a connection to the Net (and usually gets the same busies -- though customers of certain ISPs can be given a priority).
(Oh what a shock this was to your dog Wolf when first he learned of it so very very long ago... like The Net's version of "The Truman Show.")
SO... our ISP's customers may dial up some MegaPOP modem, for example, way over in Yourtown. MegaPOP answers our customers' calls and connects them to the Net.
In each town, they connect our customers to the Backbone Provider that they think is best (or who gives them the best price, whatever) in the area where that MegaPOP POP is located.
And of course, the big national backbone providers like UUNet, who also provide these bulk dialing services, will of course connect our customers to THEIR Internet Backbone, whenever possible (and profitable).
And once our customer's are on the Net... WANGO, it doesn't matter anymore WHO got them there... everything works the same... and it seems to our ISP's customers way over in Yourtown that they're on our ISP, even though they've dialed the same number to access the Net as their neighbor, who's a Gerbil.Net customer, or their great-aunt Biddie, who's an Aardvark.Net customer.
We, the ISP, pay MegaPOP or CyberPOP or some similar "wholesaler" the wholesale rate for providing our customers with dial-up service over in Yourtown; for these customers, our ISP just does the sales, the tech support, and the billing.
And we provide the e-mail and Web hosting and Usenet services (sometimes). This wholesale remote POP dial-up service usually costs us (the ISP) about $8-10/mo per account -- after some setup charges -- if we have a few thousand or more customers over in Yourtown or wherever.
NOW... Usually, with bulk dial providers like MegaPOP, we the "customer" ISP provide our subscribers with all the required extra services. But folks like MegaPOP will provide e-mail, Usenet, or Web space (for a price).
How do these folks handle user-id's and passwords?
Ok... Our ISP periodically sends to MegaPOP (via FTP) a list of new accounts, deleted accounts, and modified accounts.
MegaPOP updates their central authentication server about four times daily, which means that new accounts may take up to six hours to become active (though there is a MegaPOP Web site that our ISP can access for more rapid updates, like password changes).
Alternatively if we like, MegaPOP will send back to us all authentication requests from folks claiming to be our customers; and we then can authenticate each signon on our own authentication server. This is called "Pass-through Authentication." (Name makes sense, no? Yes.)
With Pass-through Authentication, our customers log on with something like sexy@dogwolf.net, instead of just their username. This can eliminate the delay in activating accounts for our new customers.
(Providing our own authentication could also save us $$$, because we'll then only pay for actual usage by user-id, rather than for the number of names we've put into the MegaPOP database.)
"Can an ISP like us put up its own POPs the same way MagaPOP does it?", I hear you thinking.
Sure... The typical solution is to use a frame relay connection to our remote POP, where we have a Cisco 5248 or other Remote Access Sysytem server, and then we can authenticate the logins out there via a centralized Radius server, which is just a database on some server; and of course we'll need a backup standby Radius server.
And then we'll lease rack space from some telephone company in Yourtown, and we'll install our own modems and other POP hardware to take inbound dial-in calls, and we'll route these folks to our preferred backbone supplier in that town... WHEW.
See why so many ISPs go with wholesalers? (Some ISPs go hog wild and link up with as many as five or more wholesalers.)
Let's now look at a different way of putting POPs for our ISP in another city... with a megapop (lower-case)...
Let's assume that we're an ISP and that we want to extend our service to Littletown. Ok... We lease some phone lines there, so folks can dial in; we install some modems in the telephone company exchange in Littletown; and we connect all the modems to our main facility back here at our home with a dedicated wide-band phone line, maybe a T-1.
And now, we hope with all our might that the location of our new POP in Littletown isn't too far from our home base for us to drive there in the middle of the night during a raging blizzard to fix something. Pretty straightforward stuff.
But now assume that we also want to extend our ISP's services to Bigtown, population 10 million.
Our main problem with entering the Bigtown market and offering access to all of those new customers is that MANY local POPs will be required, like say 30; because customers hate making toll calls to their ISP. They want to call local numbers.
Well, until recently, our ISP would need modems and phone lines in 30 different physical locations to support all of the folks in Bigtown. And we'd have to run dedicated wide-band phone lines (like T-1s) to each phone exchange, rent space, install modems, juggle the modems with the demand in each location... it used to be the stuff that ISP nightmares were made of.
Enter the megapop. A megapop lets us support all of Bigtown with just ONE BIG POP.
The megapop (and this can be a service provided by a local phone company) provides all 30 dial-up numbers at 30 local phone offices all across Bigtown. And customer calls to ANY of these exchanges are routed by the megapop to just ONE digital connection... so that our ISP can put ALL of its modems there, and then we can run just ONE dedicated line from that connection point to our home base, and voila... we're in business.
Because the megapop automatically routes all of our calls to one point, our ISP can set up shop in Bigtown just like we did in Littletown.
Simple, yes? Yes.
ENTER OUR COMPUTER(S)
Why do we want computers (usually called Servers)?
Well, it might be nice if our ISP could translate names like www.dogwolf.net into IP addresses like 206.191.155.8 – especially since the Net requires numbers and not names. (This conversion is called DNS – for “Domain Name Service.”)
And it also might be nice if our ISP provided some services for our customers, like e-mail and hosting their Web pages and Usenet – all cool things for our ISP to be able to do.
Add to that the fact that we might want to bill our ISP's customers and authenticate their signons, and a pattern begins to emerge.
Most functions we've just mentioned require a server (or at least a daemon, a never-ending Unix program). Do we want to host Web sites? Yes? Then we'll need a server that fetches Web pages from our disks and sends them out, in response to requests from folks. (APACHE is a popular Web server.)
Wanna serve e-mail? We'll need an SMTP server and a POP3 (or IMAP4) server. (SENDMAIL is a very popular SMTP server.) And Telnet and FTP daemons come with almost every Unix flavor, including Linux. (But be careful… these guys can open the door to security problems.)
Wanna serve up Usenet? INN is a popular newsserver. (As we mention elsewhere, be careful; Usenet is a HUGE disk hog.)
And of course, every server or daemon requires a computer (a “CPU”) to run on (though not necessarily a dedicated computer, until we have LOTS of customers).
Which servers? Well, just as an example, Sun Microsystems makes some fine, fine servers for ISPs. Perhaps for our news server we'll use a Sun Ultra 5S with 128 Meg of memory and 25 GB of disk space in an external storage pack. AND… this host could also be our primary DNS server.
(A lot of folks who sign up with an ISP like ours expect to be able to read and post to Usenet. Which is cool, except that new articles are constantly arriving in huge numbers, which can easily consume most of a multi-gigabyte disk drive. A full newsfeed now runs 20 Gig/day; some ISPs now have 200 Gig dedicated to Usenet, and most ISPs that provide Usenet dedicate a at least one CPU to handle it.
Another reason that many ISPs today are outsourcing Usenet to outfits like SuperNews is the bandwidth demand of an in house newsfeed – often a whole T-1 running full bore, 24/7.)
Another identically configured Sun server should handle e-mail and Web hosting nicely – this would allow at least 2 Meg of mailbox space and 4 Meg of Web space for each of our customers (assuming that 10% of them host Web pages). AND… this host could be our secondary DNS server. (More on DNS to come.)
Ok… Time to get our first computer for these things. We'll call it our “Primary CPU.” (Maybe we'll name it steakbone.dogwolf.net… you can see where MY mind is this day.)
Which reminds me, we'd better get that unique domain name for our ISP, like dogwolf.com. The folks who give out unique domain names (the only kind worth having) also like it if you have at least one server + a dedicated connection to a bigger ISP (or to the Net). Which now we do have.
So, our ISP now looks like this…
PHONE LINES—–> MODEMS
|
|
--------<----------------> PRIMARY CPU
|
|
ROUTER—> CSU/DSU—> T-1—> BIG ISP—> The Net
(RE)ENTER TERMINAL SERVERS
OK… so that's it? Almost. Recall we briefly discussed “Access Servers” previously? (48 ports per box, modems, digital, 56K, etc? Right.) Here we want to elaborate just a little.
Basically, we need “something” that will interface the packets coming out of our ISP's modems to our router… something that will automatically handle PPP (Point-to-point Protocol) dial up connections for us.
Well, yea… we could let our Primary CPU do it, but it's a lot more efficient to use a Terminal Server. A terminal server is a piece of hardware that connects our ISP's modems to our other ISP hardware, and ultimately to the Net. And because the terminal server (or access server, same thing) processes all calls coming into our ISP, the Primary CPU is free for other work.
HUH? Ok, let's try it again from the top… A terminal server has a LOT of connections for modems (they're called “serial ports”) and a single connection for Ethernet (called an “Ethernet Port”).
The Ethernet port connects to our ISP's local area network (or “LAN”… more about the LAN in a minute), which connects to our Main CPU.
SO… folks dial up our modems, get connected to our terminal servers, get connected to our ISP's little Ethernet network, and flow into our main computer (“Primary CPU”), which allows folks to sign in please, etc.
A single Ethernet connection to our Main Computer can handle a LOT more folks at once than if we connected every modem's output to some serial card on our main computer… Because Ethernet is a lot more efficient than “serial” protocols.
PLUS… Terminal Servers are no dummies… they're smart. If we get more than one MAIN COMPUTER (as we grow), then the Terminal Server can distribute incoming calls equally across each computer… or in proportion to each computer's power… or if one goes belly up, we can very quickly tell the Terminal Server to bypass it until it's working again.
And as we said, the separate terminal server relieves our main computer of quite an extra load that it'd have to handle if we connected our modems directly into the computer with some multiport board. Yes, terminal servers are more $$$ than serial cards, but they're intensely more cool.
And so, our reality-based ISP now looks like this…
PHONE LINES—–> MODEMS————>|
| |--------<--------------> TERMINAL SERVER | |--------<----------------> PRIMARY CPU | |--------<----------------> USENET CPU | |
ROUTER—> CSU/DSU—> T-1—> BIG ISP—> The Net
Yea, we added the second server for Usenet; lots of our customers were asking for news articles. And Usenet IS the galaxy's greatest source of information. Sexy.
Anyway, voila. (Or is it really voila?)
SECOND UPSTREAM PROVIDER
Hmmm… in all of our diagrams so far, we've shown our T-1 connecting to some BIG ISP who connects us to the Net, which is how many ISPs really do it, and which works nicely (much of the time).
But what if we connected to TWO bigger ISPs? (Recall our brief previous discussion of running TWO T-1s into our router? Right.)
Why would we want to be Multi-Homed (the technical term for connecting to two or more different bigger ISPs)?
ANSWER—–»> Because sooner or later, everyone goes to the zoo.
The biggest and best provider (or some great local provider that we've hooked into to provide our connection to the Net) can be having a really bad afternoon; or a bad week. In a word or two, if our ISP is multi-homed, we have some great redundancy.
Remember, the Net is all about who you know; it's all about your connections. We want to get packets from our customers and place them high enough in the Internet hierarchy (a bigger ISP, a Regional Provider, whatever), so that no matter what destination a packet specifies, it can get there (even it has to take a few “hops”).
Again, multi-homed costs $$$ (obviously), but our customers will see the difference in reliability and performance.
HOW ONE LOCAL ISP REALLY DOES IT
We thought it would be both interesting and instructive at this point in our journey to describe just how one local ISP really does it. This description highlights a lot of the things that a good, local ISP should feature. And it also touches upon a lot of the things we've talked about.
(We use the phrase “Local ISP” to describe an ISP with, say 5,000-10,000 customers; with 10% of these – 500-1,000 – logged on at any one time.)
This local ISP lives about half-way between Baltimore and Washington…
Introduction
This ISP charges $19.95/mo (or $17.95/mo prepaid a year in advance) with a $20 setup fee for an unlimited PPP dialup connection. The user to modem ratio never exceeds 6 to 1 (quite an excellent ratio); you can reliably dial in without a stream of busy signals. They promise, “You'll always be able to get through.”
This is a small local ISP, with fewer than 1,000 individual customers. If you pay $30/mo and a $30 setup fee, you get 5 Meg for WWW or FTP, with unlimited bandwidth. And every subscriber gets a static IP address and access to the server's Unix shell.
The ISP guarantees a response to any questions within 24 hours… usually sooner. (If they don't respond within 24 hours, you get a prize.) “We're really great people,” they advertise.
The connection between the ISP and the Internet's very high bandwidth communications lines (the National Backbone) is not so overloaded that everyone on the ISP suffers poor response.
Ok, now for the details…
Dialing In
When your modem dials up the ISP, The Telephone Company's switch directs your call to one of the ISP's 100 or so incoming phone lines. The ISP's line that you connect to is…
An available, non-busy line.
Randomized.
Why randomized? In case one of the ISP's modems is “feeling a little under the weather.” By randomizing, each time that you dial in, you'll get a different modem; because every time that you call, you'll get a different, random phone line.
If you call and hear more than one ring, you've reached a modem that just isn't working correctly for some reason. But since this ISP uses random hunting, you're guaranteed that if you hang up and call again, you'll reach a different modem.
(If the telephone office that services our ISP has the latest and greatest ESS-5 switching system, we can subscribe to the “Forward On No Answer” service for our modems… so if a line to a modem rings a few times and there's no answer, our customer's call will automatically forward to the next non-busy modem in line, who should be more willing to answer.)
A system designed by this ISP itself constantly monitors the health and status of the incoming modem phone lines, and of the modems themselves. Any failure is immediately identified.
It would be incredibly expensive and make little sense to connect each of the ISP's modems to a serial port on their computers. Instead, the serial lines go to a DigiBoard Data Concentrator. This is a RISC-based Terminal Server – and it reduces the communications load on the two main computers running UNIX, keeping them more responsive.
And the Digi-Board Terminal Server connects to the Ethernet Port on this ISP's Main Computer.
The Computers
This ISP has two fast Pentium-based computers. Each has 32 Meg of RAM and 6 GB of disk space. All I/O is done via fast SCSI-2. Larger ISPs (e.g., Earthlink, MindSpring, etc.) will have many (perhaps 50 or more) massive computers, all networked together. But this ISP does fine with just two servers.
Machine number 1 (M1) is the one the Digi-Board Terminal Server connects to. It's the one users log onto. It has all of their files, and it handles all communications between users and the Net.
Machine number 2 (M2) is a duplicate of M1. The Internet constantly feeds M2 Usenet news articles… about 550 Meg per day of articles.
(Again, note that a full newsfeed now amounts to over 20 Gig/day, which could require 200 Gig of disk space. Some estimate that by 2000, 35 Gig of disk will not hold even a day of full Usenet. Thus many ISPs limit the newsgroups that they carry, especially binaries; and they remove spam, filtering the newsfeed down to perhaps 7 Gig/day.
Almost all ISPs selectively control retention time for various groups, many retaining binary articles for only a day or two… if at all. Other ISPs simply outsource their Usenet service to a Usenet provider, such as Giga News.)
Usenet articles come to M2 from two newsfeed sources. Duplicate articles are deleted (saving about 350 MB each day). The light on M2 blinks almost constantly… the pulse of the Internet.
Both M1 and M2 run a version of UNIX called BSDI, installed from CD-ROMs and backed up on DAT tapes. This flavor of UNIX eats up almost half a gigabyte of disk on each computer… extensive and cryptic.
As it comes “out of the box,” this version of UNIX runs like molasses in Hunt Valley in January. And that's why the UNIX gurus at this ISP have tuned it extensively. It's also why we issued a warning at the outset… ISPs are for folks who know UNIX.
This ISP appreciates that loading and tuning of servers and equipment is very, very important, and that this is something that's learned only by experience. Fancy hardware, while important, is no substitute for lengthy UNIX networking experience. Driving an incredibly expensive sports car doesn't make you an excellent driver – DAwn, take note.
As a comparison, another nearby ISP, also local but larger, uses for its main computer two 60 MHhz SuperSparc CPUs. The primary machine has 256 Meg of RAM and can handle 150 simultaneous users out of its 3,500 customers; and their news server is a similar box with 128 Meg of RAM.
Returning to our smaller local ISP, the M1 and M2 computers are connected together on what is really a LAN, a Local Area Network, like businesses use to connect their PCs. At this ISP, its computers are connected together (and to their router) by a type of LAN known as an Ethernet link.
Ethernet is the most popular internal ISP networking architecture. Almost all UNIX computers (Sun, IBM, etc) have an Ethernet port, which eliminates the extra expense of an Ethernet card.
Since this ISP is using two PCs (and not “UNIX” computers like those made by Sun), each needs an Ethernet card. The two PCs and the router are configured in a “star pattern,” with an Ethernet hub at the center.
(Small ISPs use Ethernet hubs to reduce costs. The alternative is called “switched Ethernet” and provides better performance and security.)
This ISP's Ethernet network connects its three “stations” with twisted-pair wiring (called UTP, for “Unshielded Twisted Pair”) and runs at 10 Mbps.
(Many ISPs today run at 100 Mbps for scalability. Many servers – like those from Sun Microsystems – now come with 10/100 Mbps Ethernet as standard equipment – making the extra cost of 100 Mbps interfaces on the hubs (or switches) and router worthwhile.)
Our small local ISP has what is called a 10BaseT Ethernet Network.
(Yes, the folks at this ISP know that if they had everything running on one computer, and if they also had configured their router to run on this one computer, then they wouldn't need the Ethernet network.
But at the T-1 level (the speed of their connection to the Net), they know that things would run much better with multiple computers and with a separate, dedicated router.)
The ISP's LAN connects M1 and M2 to their router, in this case a Cisco 2501 router. M1, M2, and the router all speak the language of the Internet, of course… TCP/IP.
And all three of these hardware devices have different IP (Internet Protocol) addresses. (You can learn more about TCP/IP and Internet addresses and the Net itself by reading my companion article The Net by Dog Wolf.)
Why such a high quality and expensive router as the Cisco 2500 series? The router needs to examine all the packets on this ISP's internal LAN, locate those that need to be routed to the Net, and send them zipping along on the ISP's T-1.
It's not rocket science, but the router has to be able to operate at very high speeds, and a Cisco 2501 can handle the load. A router that can't handle its load can be the cause of catastrophic errors, and so this ISP has opted for a high quality router.
(If they had a significantly slower line than a T-1 connecting to the Net, there are hardware boxes that combine the router with the CSU/DSU. They understand, however, that these will not perform well, as their load (and speed) increase.)
Onto The Internet
When you send mail to a friend, connect to a Web server, or do any of the other wonderful things available on the Net, computer M1 at the ISP sends your request to the Cisco router. The router is really a smart bridge, linking the ISP's LAN to the Net itself.
Now… this ISP probably knows that it could use a PC running some flavor of UNIX (perhaps LINUX) for its router, perhaps even use part of its main computer.
But it also knows that this could raise some serious security and performance issues. And so they've invested about $2,000 in their Cisco 2501 router, and it just sits there in the rack, and it hums, and it routes… never a problem. These folks know that the Cisco 2501 can handle routing up to T-1 speeds very nicely.
This router has only one port where you can connect to an Ethernet LAN (this ISP's internal network), plus two serial ports, but that's all this ISP needs… an interface between it's internal Ethernet LAN and its T-1 that heads out toward the Net.
Should this ISP grow to the point where it has more than one internal Ethernet LAN that it needs to connect to the Internet, they'll probably upgrade to one of the more expensive Cisco 7000 series of routers, which has ports for multiple Ethernet LANs.
But for now, with fewer than 1,000 users, and with a single Ethernet LAN and a single T-1… the Cisco 2501 router works just fine.
(General rule of thumb for ISPs… if it's made by Sun Microsystems or by Cisco, it'll be a bit pricey, but it'll be very reliable; and (if you've sized it properly), it'll do its job well for a long time.)
The output of this router goes to a Kentrox CSU/DSU, a kind of ultra-high speed modem for the ISP's T-1 Internet link. Many folks will insist that Kentrox is the best CSU/DSU available, despite its price tag – usually over $1,000 for a CSU/DSU that will connect to a T-1.
(CSU/DSUs that connect to slower lines, like a 56k line, are about half that price.)
The folks at this ISP think of a CSU/DSU as a sort of “digital modem” that they have to have when connecting to a phone company leased digital line, like their T-1.
The CSU/DSU connects to a T-1 telephone line, which can pass data at 1,554 Kbps. The T-1 is a copper line with repeaters for the first two miles to the Telephone Company's nearest central office. Here, it converts to 16 miles of fiber optic cable that run to Baltimore.
This ISP might have considered a frame relay leased line instead of a T-1. A frame relay line connects you to a network operated by the telephone company, which can connect you to bigger ISPs located in distant cities.
But a frame relay T-1 is not as fast as a true T-1; and so this ISP has decided to go with the real thing.
This ISP intends to stop taking new customers when it reaches 1,000 users. As a result, the T-1 will never service an excessive number of users. (There are ISPs out there with 10,000 individual dial up customers with just a single T-1 connecting the ISP to the Net… see why price is not the only consideration when selecting an ISP?)
The T-1's fiber optic cable runs into the City, where it links to the Internet Backbone through another router and CSU/DSU.
(Note that some large ISPs through which smaller ISPs gain Internet access require the smaller ISP to provide the CSU/DSU and router at THEIR end of the T-1 also; and others will provide the hardware at BOTH ends. Obviously, this is reflected in the charges.)
This local ISP's T-1 is not the simplest or cheapest way to connect to the Net, but this ISP went for a rather “top of the line” system to guarantee that users would not see see too many bottlenecks.
And that's how one small local ISP really does it.
JUMP TO BIG
Now let's look for just a minute at how MindSpring does some of this same stuff, with over a million mostly non-commercial customers.
MindSpring has standardized on 1,000 Megabit (called Gigabit, or GbE) BigIron 8000 Ethernet switches from Foundry Networks to replace their older Cicso and Bay switches.
This will support their growing Web hosting business and a user base that has been doubling every few months.
In addition to the explosive increase in e-mail and Web usage, the new Gigabit Ethernet switches can support xDSL, streaming video and audio, and cable modems.
The Bigiron model 8000 has 64 ports of Gigabit Ethernet in a single chassis (at almost $2,500 per port). It has a switching capacity of 256 Billion bits per second, and it can handle up to 100 million packets per second.
(MindSpring also has an 8000 switch in its new Phoenix tech support facility, which will eventually have 500-600 workstations.)
At MindSpring's center of ISP operations in Atlanta, there is one 8000 switch, with 64 ports -- all running at 1,000 Million bits per second. This site houses the company's production server LAN, which includes nearly 90 Digital Alpha servers and Network Appliance filers as its disk farm... which support e-mail, Web access and hosting, Usenet, and FTP.
All the servers will eventually migrate to Gigabit Ethernet, as will MindSpring's current Cisco 7513 and Bay BCN routers.
MindSpring also plans to upgrade its POPs from 10 Megabit Ethernet to Gigabit speeds, once it deploys cable modems and xDSL services.
Ultimately, the company expects to have identical systems in its Phoenix and Harrisburg, Pa tech support centers; and there will be a "server farm" in Seattle which will mirror the Atlanta site.
And that's how one nationwide ISP really does it.
HOW ISPs REALLY CONNECT TO THE NET
Now your dog Wolf wants to talk a bit about how things connect together on the Net. Remember, it's all in who you know, your contacts.
Up until now, we've avoided this BEEEG issue entirely by muttering something about our ISP simply connecting to some “Bigger ISP” via a T-1. We let Bigger ISP worry about connecting to the Net.
(And this is exactly how a lot of smaller ISPs DO do it.
A small ISP WILL connect to some bigger ISP who connects to the Net, to some "National Backbone," as we'll see in uno momento... or dos. OR... our "Bigger ISP" may instead be some Connection Service, like, say, Los Nettos in Los Angeles.
We, the small ISP, connect to him, and he (the big, manly, robust connection service) makes the actual connection to the big, high-speed cables that criss-cross the world and carry most of the Internet's packets.
These connection services -- like Los Nettos -- are also a cool place to get the IP addresses that we'll assign to our customers when they log onto our ISP... more coming on that later, mates.)
But now, we're going to have to face reality; and like salmon, see exactly WHAT is upstream from us…
The Net is just like a lot of cars running along roads and highways… The cars are packets. Each car sports a large sign saying where it would like to to end up. (“Key West or bust,” remember?)
Let's call our longest, fastest, and widest highways “Interstates” (since that's what most folks call them). And we'll have our Interstates cross each other or terminate at wide, high-speed “Beltways.”
So, we arrive in Houston on I-10; we drive around The Loop (I-610) till we come to I-45; and then we leave Houston on I-45. (Yea, your dog Wolf gets around.)
Now… the Net's highways are especially cool, because our circular Beltways have friendly (smiling) traffic cops stationed about them. We don't even need to know which road to take out of Houston.
The friendly cops read those signs on our cars (“Key West or bust”), and when we come to the correct highway to take us toward Key West, they wave us onto it. Cool, yes? Yes.
On the Net, the Interstates are called National Backbones. The Beltways are called NAPs (Network Access Points). And the friendly cops are called (you know this one, right?) Routers.
As dog Wolf counts them, there are 12 Beltways in the US section of the Net (other folks may count only 11). Some, like the NAP called “MAE-East+” in Washington, D.C., actually circle the city – just like the infamous and incredibly frustrating I-495.
Other NAPs are just big equipment rooms, where phone lines from AT&T and Digex and other National Backbone Operators connect to very fast routers.
Ok… So exactly what are these National Backbones that operators like UUNet have built, these “Interstates of the Net?”
Take some fast routers and put them in big cities about the US. Connect these routers with fast phone lines… perhaps fiber-optic, ok?
Between some cities, even string TWO connections, supplied by different phone companies (just in case… called “redundancy”). And then connect these massive Interstates, bulging with packets, to some of the NAPs, the Beltways.
And look… you have just spent $1 billion and created another National Backbone for the Net, exactly as AT&T did recently.
(Some NAPS are created more equal than others. The FOUR official US NAPs are located in NJ, DC (the infamous MAE-East), Chicago, and San Jose (the equally infamous MAE-West).
Now, let's examine what happens to I-95 (the automotive National Backbone that runs from Maine to Florida) when it crosses the Delaware into New Jersey… divides into lots of roads, doesn't it? (Stay alert.)
But mainly, it divides into I-95 and the NJ Turnpike, both heading northeast toward New York City. And for many miles, these two competing roads run parallel, side by side, perhaps a mile apart at the most.
Now assume that you start out on the NJ Turnpike, but you hear on the radio of a jam-up ahead. (Actual case history… happened to DAwn when she was returning from a wedding one hot Sunday afternoon last August.)
You'd really like to get over onto I-95. Do you REALLY have to struggle along on the NJ Turnpike until you come to some Beltway? Nah… you get off the Turnpike at the next exit, take a short connecting highway for less than a mile, and enter I-95.
The identical thing happens on the Net. Here, it's called “Hot Potato Routing.” I like to think of it as, “Get it off my plate as fast as possible and onto someone else's” routing.
An example… I'm in Houston, and my PC is connected to MCI's National Backbone. I send out onto the the Net a packet that's destined for some Web server in San Diego, a server that's connected to SprintNet.
Does my poor packet have to go all the way up to the nearest NAP in Chicago to peer (connect) with SprintNet?
Nah… MCI looks at my packet's destination, sees that the destination server is connected to SprintNet, and dumps me off onto SprintNet in Podunk, Texas, where MCI just happens to have a Cross-Connect to SprintNet.
Once the friendly traffic cop at the Podunk, TX, router waves my packet onto SprintNet, it'll travel pretty directly on to San Diego along the SprintNet National Backbone.
MCI… he just dropped me like some Hot Potato… got me off of his plate and onto SprintNet's… less traffic on MCI's plate, more traffic for SprintNet's. ¿Está claro? Yes.
These National Backbones peering (connecting) at the NAPs and Hot Potato-ing with each other is the GUTS of the Net. And the Net has more and more guts every day. Ok, onward… where do ISPs enter the picture? Stay tuned.
Now think back on how we created a National Backbone. We put big routers in big cities and connected them with fast phone lines.
But lots of OTHER things can connect to these National Backbone routers in the big cities… in fact, that's the reason that folks like AT&T build them in the first place… you don't build an Interstate without installing Interchanges where traffic can get on and off easily. (Hint.)
At the routers in the big cities, the National Backbone Operators like UUNet and AT&T will allow you to plug things in, so that you can put your packets onto their network, and so that you can take your packets (packets with your destination ID) off of their network.
Yes, there are on- and off-ramps. But they're toll roads. For the privilege of connecting to the National Backbones at their routers, you will pay them $$$ (money).
So out of the fast National Backbone routers in cities come lots of other phone lines. (Lots of roads of all sizes want to connect to the Interstates.) Some go to towns, some go to small towns.
The UUNets and SprintNets run T-1s and 56k leased lines and whatever folks are willing to pay for from their big, fast, manly routers to their POPs (Points of Presence).
Here, others may connect to the National Backbones. And these others are usually Businesses and Universities and Government Agencies and… and… ISPs. Yes. (Whew.)
And in fact, for a price, a National Backbone Provider may even let our ISP's customers connect to the other POPs that he has located around the world… T-1s, 56k, even dial-up modems. (Recall our discussion of MegaPOPs and remote POPs – here we've just approached POPs from the other end – from the Net.)
And this is how many small (and large) ISPs manage to have POPs all over the place… and why, in the case of dial-ups, 15 different ISPs have the same phone number in Podunk, TX… because they're all using the same UUNet POPs. (Yea, he knows how to keep the packets separated.)
This is why many of the IP addresses for GTE's dial-up customers translate into names that end in something like branson.mo.da.uu.net. (Think some GTE customer in Branson, MO, dialed into a UUNet POP? Looks that way.)
ISPs connect to the POPs on National Backbones through leased lines. In turn, businesses will connect to ISPs through leased lines, and they'll provide Internet access to their employees via internal networks (usually LANs) connected to these leased lines… in effect, the businesses have become ISPs, with their employees as the customers…
NAP#1 NAP#2
| | | (<---National Backbone--->) | | |
~~~Fast Router~~~~~~Fast Phone Line~~~~~~Fast Router~~~
/ | | | | \ / | | | | | \
|
|
POP~~~~~Leased Line~~~~~*ISP*
|
|
Business~~~~~~Leased Line~~~~~~POP
|
LAN
/ | \
Employee
(Think this could be one reason why the Interenet is so pervasive today? Uh huh.)
Moving down one more step in the hierarchy, we encounter the State Highways, each of which may actually traverse a few states. On the Net, these are called Regional Networks. They are similar to the Interstates, the National Backbones; they simply service smaller areas… often just one state, perhaps a few states in a region.
And just as the State Highways access the Interstates when they can, a Regional Network will connect to as many National Backbones as it can. Hey… it may even connect to a Beltway (a NAP).
Like our Interstates (National Backbones), the Regional Networks also sprout leased lines at their big city routers – which also end in POPs. And ISPs can connect to these POPs through their own leased lines.
UNDER THE COVERS
Yea, we know... this is how it works MOST of the time... but the times, they are a-changing, to use an old Aussie homily.
In order to avoid the congested NAPs on the public Net, a new type of IP packet carrier is emerging. These new carriers buy "transit" services from several of the National Backbone providers. ISPs connect to these new guys, and the new guys give them access to the National Backbones.
WHY? The new guys' architecture keeps their customers' traffic out of the public NAPs, out of the public peering points. Because, you see, the NAPs are VERY busy these days, especially at rush hours. And busy means packet loss and congestion and retransmitting lost packets = SLOW.
The new guys bypass the NAP bottlenecks by passing traffic directly to the National Backbone providers... in effect, they have their own private on and off ramps to the Interstates... and they don't go near the Beltways. (Ever been on one of the major metropolitan Beltways at 515 PM? Like I-280/ I-880 in the Bay Area? Not so cool.)
Why don't the ISPs just do this themselves... buy private access ramps onto the Interstates? Well, they surely could (and some may), but these new guys save ISPs the expense of buying several separate "transit" connections.
Basically, the new guys are buying wholesale fiber optic cable from local and long distance phone companies. They then sell multiple paths to the Internet to ISPs for less than the ISPs could buy it on their own.
And the bottom line here is that ISPs may enjoy better performance. One of the "new guys" is a company named Vaultline.
Vaultline has some really cool routes to the National Backbones; and if one of their connections fails for a while, an ISP can send a packet through another -- because Vaultline has direct access to FIVE National Backbone providers.
Sorry to complexify things with this nugget, but hey... this is how folks are doing it in the real world; and this is surely something that our ISP wants to be aware of.
This is the most common form of ISP connection to the Net today…
An ISP runs a leased line (perhaps a T-1) to the POP of some Regional Network.
The POP connects via a leased line to one of the Regional Network's routers.
The router passes packets to and from the Regional Network's high speed lines.
The Regional Network's high speed lines connect to several of the National Backbones (and perhaps even to a NAP)... And voila... you're on the Internet.
It ALL looks like this…
National Backbone #1
~|~~~~~~~~~~~~|~~~~~~~~~~~~|~~~~~~~~~~~|~~~
| | | | | | | |
NAP#1 NAP#2 NAP#3 NAP#4
| | | | | | | | | | | |
~~~~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| National Backbone #2
| /|\ /|\ /|\
| POP
| |
| |
| ~~~~~Leased Line~~~~~*ISP*
|
|
~~~~~~~~~~~~~~~~~~~~~
A Regional Network
/|\ /|\ /|\
POP~~~*ISP*
DAwn's Right Eye Your suggestions, corrections, updates, clarifications, additions, and general and specific comments on this page are intensely encouraged. Please send your comments, remarks, corrections, criticisms, flames, greetings, threats, messages, love-notes, and whatever to DAwn. All feedback is INTENSELY appreciated.
[This text and its HTML representation are copyrighted by Dawn McGatney and other authors; this Web site constitutes this work's publication under Law, in accordance with US Federal Law and applicable international agreements. This work may not be reproduced in any manner without the express written consent of the authors. Any unauthorized reproduction constitutes a violation of copyright. External hypertext links appearing in this article are copyright of their respective authors.]