Wednesday, October 21, 2009

How the Lab Exam Should Be

I have just done one lab exam to complete the internal certification from my organization. I can’t disclose more information about it, but what I can share here is my thought of how the lab exam should be done based on my experience taking that exam.

I said it once that certification means nothing without experience. Passing even a very tough lab exam such as CCIE doesn’t turn us to become real expert directly. Certification can only offers the baseline set of skills, and we should build our expertise on top of these skills, not in lieu of them.

But have you ever wondered how far is the skills tested in the lab exam compare to the ones needed in real life? For example, once you pass the CCIE SP lab, do you think you can just jump into a large SP environment or there is still a huge gap to fill in first?

Let me share my thought of how the lab exam should be done. As usual, this is just my personal opinion. And you know what they say; opinions are like arseholes, everybody’s got one.

1. Lab exam should use the real gear

If you look at the equipments in CCIE SP lab, you will notice that they are not the real Service Provider gears. Cisco 7200 is good, but not as P router! As well as the 2800 and 2600 that are still being used in some lab.

SP lab exam should use high end routers such as CRS, ASR and 7600. As P node, if multiple CRS is considered too expensive, a single CRS with physical partition using Secure Domain Routing can do the job as well. And we all know most of the SP core networks use IOS XR, so at least GSR with IOS XR must be available if getting CRS is not an option. For PE node, if the latest ASR9K is still out of reach at least use 7600 with RSP and ES+ card!
Sound too ambitious? Perhaps. But those are the real equipments being used in most Service Provider networks nowadays.

It’s the same case with CCIE Routing & Switching lab. If this track is supposed to simulate a large Enterprise network, at least Cisco 6500 should be available in the lab.

2. Lab exam should simulate the real scenarios

Okay, you are done with the configuration of the device, and then what? Run a ping test? Verify the config? Run the show commands? It’s not enough!

The lab exam should use traffic generator to simulate the traffic. Once we have the traffic in the network we can verify, for example, if the Quality of Services features really work. The lab should ask the candidate to verify the failover scenario. How we can be sure if the fast convergence feature is already configured properly? By checking the BFD neighbors from show commands? By looking at the NSF and GR config only? Yeah, right.

Why can’t we just run the traffic generator and see the impact of the configuration, or failover scenarios, to the traffic? Even the skill to understand and set the traffic generator is necessary to do the job in real world later on.

3. Lab exam should test the knowledge in proper way

It’s not enough to ask the candidate to configure or troubleshoot something in the lab. Some guys can just get the lab questions from somewhere and memorize the configuration to answer them.

The best way to test the in-depth knowledge of the candidates is by asking them to do the verification and explain the output. For example, during fast convergence test, let’s ask the candidate to provide the convergence time for link failure and ask them to explain why the time can be different between link down and link up (restoration) state. Can they explain why the convergence time can be different if the PE router crashes compare to if the failure happens in P router?

Tricky questions like in current CCIE lab are still important. Troubleshooting skills are still required to be tested in the lab too. But the candidate is expected to be able to explain more ‘WHY’. Not only why it’s configured this way or that way, but as well as why the traffic behaves in certain way when some features are configured or when the failure occurs

I know most of the time it’s unfair to do the comparison between my “ideal” lab exam with the known certification like CCIE.

Take the lab equipments, for example. The real gears aren’t cheap. So a vendor may have only 1 or 2 complete labs that can replicate the real world’s equipment to serve all candidates around the globe. My view on this: that’s fine. Because nowadays we don’t have to fly to sit in the lab physically, we can just do the exam remotely. And the lab exam I described above is to test the skills in specific area. I mean, Cisco may create “Advanced CCIE lab” for specific technical focus with CCIE as the prerequisite, and there are so many tracks available (CCIE SP-NGN, CCIE SP-IPTV, CCIE SP-Wimax and so on). With many options of advanced track available, a candidate can choose which one is suitable to support his daily work so the number of candidates will be distributed to all the tracks.

If time permits, all the explanation should be done with short interview, not only in written. How if English is not the native language of the candidates? That’s fine. With remote lab, more locations can conduct the exam and the candidate can have the proctor who can speak the same language. And the lab exam can take 2-days format just like back there in 2001. Day 1 can be allocated to build the network, Day 2 morning can be used to run the traffic generator and verify the setup. Day 2 afternoon is for troubleshooting section. At the end of each section the candidate is expected to explain what have been done, and the behavior of the traffic in several different scenarios.

Obviously with this new and advanced CCIE track the main objective is to prepare the candidates to able to do the job the next day after they pass the lab, and not to chase the quantity of people to pass it.

Is it possible to have that kind of lab or can it be done only in our dream? Heck, who knows? One day a vendor like Cisco may really create a new certification track beyond CCIE, and they may take all the points above into their consideration.


Scott Morris said...

While I agree about the idea of making things real-world, I don't see the need to change out all the equipment. Let's pick on your example of SP-NGN. What are we going to do if you have 7600's in the lab? They don't support multipoint-LSPs anyway, so we're stuck on GSRs and CRSs. $$$$

Gets back to the point of what are you tested on? Technology. If you can do TE on a 2800, can you not do it on a GSR? If you can do QoS on a 3600, can you not do it on a 7600?

IOS-XR is different, yes. There are some features that may be nice to know but not all SPs run XR devices. So there, the argument is backwards. Why am I learning this when I'll never care?

It's a technology exam. You learn the stuff, you fill in extra details later. R&S didn't teach you how to use a 6500 or any service modules. Do you care? Or can you expand upon what you know and go higher?

If you get a PhD in something, do you know everything? Or are you minimally qualified at a baseline with the assumption you'll go further?

Wake up. Move as far ahead as you need to, but your baseline hasn't changed. Until the world moves into all that gear (or we could put in 3750-MEs just to piss everyone off with the ES ports) then there's no point.

As it stands, the one thing I think WOULD make sense is IPv6. Amazingly, that's not even looked at. The R&S has it for god-knows-what-reason, but the SP track doesn't.

Just my thoughts.

Himawan Nugroho said...

Scott, have you ever compared what 12.2S in 7200 can do with the SRD-SRE in 7600? And I bet you know it for sure 7200 doesn't run VPLS :)
Having all XR and ASR9K/7600 in the lab won't make you the real expert either after you pass, but at least the gap is not too far. That's my point.

Anonymous said...

I think you have completely shown your massive ignorance of certification and your arrogant lack of intelligent understanding by this post

The challenge is not the technology ..packet in packet out THIS WILL NEVER CHANGE

The challenge is the aptitude and attitude ... the PERSONS ability to change and keep up with technology in general is key

time to null route your A record and call it a day .

Himawan Nugroho said...

Ah, sweet music... thanks for such words even I prefer to hear it from someone with name... Perhaps you skip the part when I said 'opinions are like arseholes..' or perhaps you take it literally?
I have passed 3 CCIE labs and bunch other certifications from another vendors. The last lab exam I passed was CCIE SP. And I've been handling SP customers from APAC, Eastern Europe, Middle East and Africa. All that I wrote is based on what I have seen and been through. The coverage of CCIE SP, for example, is just not enough to prepare the ones who pass it to handle NGN network. And I know the limitation is due to the hardware being used in the lab. Cisco 7600 or the new ASR can really change the game as PE nodes.
I agree the attitude and aptitude are important. But many guys are not that lucky to have high end routers to play at home or in their current job, so this advanced lab exam is the opportunity to test and implement the concept they can only read before. Not all SP run IOS XR so why bother to learn? Well, most of large scale SP run either XR or JunOS, and if you know either one it's easy to switch. But again, hey, it's just my opinion. And I have seen many SPs face the shortcoming in the operation team's skills to handle the NGN infrastructure. Even this means good business for service company like Cisco AS, I still wish there is a way to shorten the gap.

Anonymous said...

>> have passed 3 CCIE labs and bunch other certifications from another vendors

I think this speaks volumns !!!!
Oh im a king what I say goes
where teh fuck do you come from ?

Folks just becuase you pass some exams doesnt mean that your opinion counts ..
I dont give a shit whether you graduated from MIT OR MIB

argue your points based on the merits of the situation and show your intellect ..if you have any

your just an arrogant noise that indulges in your own self inflated opinion of yourself

Oh do you work for Cisco ..that explains it then
Cisco the place where every engineer kisses every other enginers ass ...your so great your so wonderful ... mmm mmmmm

grow the fuck up and put some hair on your balls

Himawan Nugroho said...

I will keep your comments here just to remind me of how some people just can't communicate properly... but any further comments that can only cursing and bitching, especially from those who like to hide behind the anonymity, will be removed...

Unknown said...

What's the big deal with being anonymous?

So what if someone changes from "anonymous" to "Dave Thomas", does that make a difference?

The saying goes "Opinions are like assholes; everyone's got one and they don't think theirs stink!"

Himawan Nugroho said...

Hi dragons, thanks for your comment... there is nothing wrong with being anonymous... but if ones want to curse, use the F word and accuse someone or some organization, in my very own blog, then I would prefer for them to speak their name out loud.. else, just do it somewhere else :)

Yes I work for Cisco but using this blog I just want to voice my concern regarding the CCIE and CCDE exam format (already sent the feedback directly during the exam). I still respect those who put their dedication and hard work to create such exams, so I believe there is always another way to communicate our opinion then cursing...

Anonymous said...

Classic arrogance.

Maybe rather than dictating what CCIE should be for all of us...

You should concentrate on Family.

Scott Morris said...

Wow.... Well... That was an interesting digression.

In general, having different opinions is part of what makes the world go 'round! However, if you'd like people to listen to you, you may consider coming off a bit less abrasive (to the anonyous clan).

Nothing wrong with being anonymous, but a lot wrong for not being professional.

The number of CCIEs doesn't make or break someone as a good engineer. But an idiot working on the coolest equipment/networks in the world who doesn't understand base fundamentals is apt to only squeak by instead of being able to grasp everything.

Packets in, packets out. Blah, blah, blah. But if you don't have all the bandwidth in the world (even the largest of the SPs I work with have limitations!), or an unlimited budget, it's much more than packets. If you don't get that, well... Hmm... Guess ya just don't get that. ;)

Since I'll keep this spirit of being abusive (but professional) to everyone... Yes, I know well the differences between 7200 and 7600... Also painfully aware of differences from 7600 to GSR. :)

Now, since you mentioned VPLS, please tell me what there is that's technically difficult about configuring VPLS versus the more generic pseudowire connections the 7200 will do?

Life doesn't really get interesting until we get to do p2mp lsp's, but the 7600 doesn't support that yet anyway. So GSR? Point being that I (and many others) who have their CCIE SP and were never tested on some of those cool things, are perfectly capable of picking it up.

Because we have a sound foundation to build upon. It's not learning a brand new thing with VPLS (or insert topic here). It's about learning what's different from things you already know. It's about adapting. Some can do it and some cannot. But latest and greatest (or avoiding the "features" aka bugs) is not part of the lab testing philosophy. It's the reality that will occur there.

Med school doesn't prepare a doctor for everything. but it's a fundamental baseline and a thought process for later expertise.



Himawan Nugroho said...

appreciate your comment, I prefer to read yours many times even if it's not quite similar with my thinking, compare to other comments from guys who like to accuse and use inappropriate words...
Perhaps I'm supporting the wrong SPs, since the ones I've been dealing with really concern about the fast convergence, using IOS XR in the core or the ES+ card in 7600 where there are not only new interesting features but as well as the way the feature works can really make a difference...
Yes, adaptability is very important, I completely agree. But to be frank, I believe the only reason why the SP lab is still using old gears is just because of the funding, especially since the demand is not as high as R&S lab. One day the hardware will be upgraded for sure, probably to GSR-XR and the ASR 1000 at least, then we will see more things to come for this track...

Himawan Nugroho said...

Ah, some of my previous comment may give a wrong impression... the reason I said "I've passed 3 labs etc" is not to show how great I am and all you mortals hail me, but just to point out on how do I reach the conclusion that there is a gap between the lab and the real network (at least with the SP customers I've been dealing so far). Especially in the part of the world where I live now, there are so many CCIEs, even double CCIEs, and I know some of them really worked hard and shed their blood to pass the lab, only to find there is still a huge gap to fill in first before they can handle the SP network properly. Natural selection? Only the strong will survive? Probably. But if there is a way to shorten the gap, why not? That's the reason why the R&S lab keep evolving, isn't it? Unfortunately for SP lab, I guess due to the less popular demand, the hardware upgrade or the change of the format won't be that often...

Scott Morris said...

There will always be a gap in skills. That's true no matter which track we are talking about or which "reality" of customer we are talking about.

I still remember passing my first CCIE back in 1999 (way back with all the "exciting" desktop protocols) and getting a consulting gig regarding DLSW afterwards. Now, I'd always hated DLSW, but I figured "I'm a CCIE now, I should be good!".

The gig wasn't really DLSW. It was about STUN tunnels and BiSync stuff. Loosely touching a DLSW network. Things I had ZERO experience with. But on the other hand, I had enough background, and enough conceptual thinking to make it a successful venture. Would I consider myself an expert at Serial Tunneling? God no. And yet it still worked.

The CCIE (or any certification) honestly only establishes a "minimally qualifying standard" of knowledge. It's not a solve-everything-in-the-world exam. There are things on ES+ cards or whatever (insert specifics here) that people won't know. There's likely several things even I do not know despite consulting and training background.

But the question is could I learn it and deal with it in short order? The answer is yes. That adaptability isn't really tested on, but THAT is what makes someone a good engineer. Experience/knowledge is all relative.

The "genius" in one area may well be a "dunce" in others. But yet can still adapt. A new doctor hasn't experienced everything, yet we trust their ability to figure things out for us no matter what the issue.

Budget isn't part of the picture IMHO because the SP-BU offered to give (free) the CCIE team a bunch of GSRs years ago and they turned it down.

It's a more academic approach. There are a few things (VFI, etc) that may be interesting to have, but not critical. If you can think through the LSP process for a pseudowire, you can think through it for VPLS.

Good sparring. ;)


Eugene Han CCIE #10890 (Security, RS, Service Provider) said...

Scott, I really like your analogy of the doctor. It really hits the nail on the head. Himawan, you have some very insightful observations... this website is actually pretty accurate in terms of the CCIE personal evolutionary path. I am a contractor who specializes in nyc wall street financials. I like what the Cisco Certified Architect exam has introduced into our world. It will weed out 99% of the ccie's out there.. the weak.. the ones who will scatter into the shadows when the bright spotlight is turned on them.. Cisco's personal stamp of who the best in the world are via real-time board exam style interaction. A true capture the flag, king of the hill situation. I love it and embrace the challenge. I'm going for my CCDE second attempt in Chicago. Good luck on the CCDE Scott and Himawan. See you guys on the other side :)

Himawan Nugroho said...

Hi Eugene, I'm glad you brought up this discussion again :) yes, Scott has a valid point.. and even med school would upgrade the practice lab in the campus with newer equipments :) take care and good luck with the exam, it's about 14 hours before the CCDE here in London and I haven't prepared anything thanks to the project... c'est la vie ;)

Scott Morris said...

And now it's back to the "waiting time" between the CCDE Experience and the grade report. :)

Good luck to all who attempted! (I'm not convinced preparation would help!)

Anonymous said...

Just guys have 5-6-7-8...CCIE/CCDE certifications and you can't deny it, you are proud of them :))

How would it be if you would have to re-certify each one of them in particular ?
How about the certifications from other vendors? What if you have to recertify those too?

How come that none of you is arguing with this hypocrisy or recertifying Voice with a Storage exam let's say.

I am not respecting you less for that but I wouldn't be myself if I wouldn't ask you this questions.

When I am looking at your achievements and glory half of my face turns green, I envy you, the other half is smiling amused because my mind flies to this story:

One man lived his entire life with one target in mind: to become the holiest human being and to have his name chiseled/engraved on the wall he knew it existed in Heaven for this matter. When he died he got there and the guardians of the wall handed him over the chisel and and asked him to put his name there himself. He couldn't find a free inch not even for the first letter of his name. The wall was already fully covered by names.