Middle Kingdom LifePerspectives on Living & Teaching in China
Evaluation System for Foreign Language Expert
Middle Kingdom Life recently received an inquiry from a foreign teacher regarding an online “psychological test” he was asked to take by a prospective employer as a pre-employment screening tool. We had never heard of this test so we scouted around on the Internet until we found the “Evaluation System for Foreign Language Expert.”
The evaluation system consists of five distinct parts: 1) Evaluation of the candidate’s posted résumé; 2) A Psychological Test; 3) Elementary Test; 4) Writing Examination, and; 5) An Online Interview which must be scheduled in advance. Finally, an evaluation report is available to the registered user so that he or she can see the numerical results of the three online exams (presumably with a maximum score of 100 on each subtest).
Before discussing the substance of the three online exams, it is worthwhile to note how the administrators describe the purpose of this evaluation system:1
This system has been developed together with the State Administration of Foreign Experts Affairs, PRC and the Capital University of Economics and Business. It is designed to carry out the recently promulgated Administrative Licensing Law of the People’s Republic of china. The Law is being instituted in order to regulate the issuance of permits for foreign experts and to resolve current problems on introducing oral English teachers into the nation’s public education services. In a long-term perspective, it will accelerate the building the international professional personnel market, foster and standardize international personnel agencies and further strengthen market access, regulation and supervision.
‘This system consists of three subsystems: resume filtration; written testing, and an online interview. The tests will comprise a general personnel evaluation psychological profile and a general knowledge test on China and other current world topics. This system will automatically analyze these tests and generate an evaluation report. The results will help employers evaluate the candidate’s basic information and his/her abilities to become a successful professional worker in China. It will also offer candidates a reliable measuring tool to know their strengths and weaknesses and give a more accurate career orientation.
Evaluating the Evaluations
For the purposes of this article, I registered a user account and then proceeded to take each of the three online tests. Before answering each exam, I created digital snapshots of all the questions for further analysis. As the material is copyrighted, I cannot post the exam material on this site but, as you will soon learn, it is entirely unnecessary for me to do so for the purpose of preparing those who might be asked to participate in the evaluation. That is, based on what you will read, you will very easily be able to score quite high on these exams. And therein lies the first of many problems with this evaluation system.
The manner in which the exams are administered renders them completely useless (assuming the exams themselves were valid, which—as you will learn—they are not at all). Aside from the fact that there is no practical way to assure that the registrant is in fact the exam taker, one can create digital snapshots of all the questions as I did, look up the correct answers on the Internet, and then re-access each exam after a 24-hour wait period.
The Psychological Test
I started with the psychological test first as this was the one I was most interested in (examinees can take each of the tests in any order). It consists of 100 questions and presumably seeks to measure 23 distinct personal attributes or characteristics:
- Adaptive Ability
- Drink
- Extent of Mental Health
- Harmonization Capability
- Go-aheadism
- Heart Endurance
- Confidence
- Motivation
- Communication skills
- Sense of Achievement
- Expressive Ability
- Emotional Stability
- Tendency Towards Violence
- Tolerant Capability
- Inspiring Capacity
- Professional Spirit
- Logical Thinking Ability
- Comprehensive Analysis Ability
- Sense of Responsibility
- Organizational Ability
- Optimism
- Face Tendency
- Response Ability
Upon a cursory examination of the 23 traits this test purports to measure, one need not be a Western psychologist to accurately surmise that this is neither a standardized test nor one that has any precedence of use. Many of the questions are loosely based on items that can be found in classic personality measures such as the Myers-Briggs Personality Inventory, but the items are seemingly and deliberately reworded, and quite poorly at that, to avoid copyright infringement I presume.
For at least four or five questions, I had to guess at what was intended and that alone invalidates the test, i.e., a test cannot accurately measure what it purports to if the meaning of the questions is unclear.
It was obvious to me that the instrument was written to measure what the SAFEA believes would constitute a good foreign teacher: someone who is resilient, cooperative, takes initiative while at the same time is able to go with the flow and adhere to the will and direction of the majority, is a non-smoker as well as a non-drinker. The test suffers enormously from demand-bias, that is, it is extremely easy to ascertain which answer is the “correct” one (especially after reading this article). I made the mistake of second-guessing the expectation of the exam’s authors in regard to cigarette smoking: Surely, I thought to myself, in China, cigarette smoking must be considered a desirable trait? Because I deliberately padded my responses to the four questions on cigarette smoking to indicate moderation (as opposed to strongly rejecting the behavior), I scored a 93 on the test instead of what I assume would have been 100 if I had strongly endorsed all negative statements about smoking.
The answers to each question are measured along what is intended to be a standard 5-point Likert Scale, which was anything but conventional: Highly Accept—Accept—Moderate—Not Accept—Refuse. I personally found the last category of response to be very confusing as I wasn’t initially sure if “refuse” meant that I was refusing to answer the question or that I was indicating strong disagreement (based on my final score, it is fair to assume that the latter response was intended). Why the authors of the exam decided to use such unconventional labels for recording responses when years of research have validated the accuracy of simply using “Strongly Agree, Agree, Undecided (or ‘neither agree nor disagree’), Disagree, and Strongly Disagree” was completely lost on me.
Finally, many (if not most) of the questions are poorly translated into English such that there is one typo (”wist” instead of “fist”), idiosyncratic word usage, and several unclear meanings. I consider this test to be neither valid nor reliable in regard to what it claims to be measuring (much like the CET and TEM exams, see below). The exam is timed at 20 minutes, presumably to force the examinee to answer as spontaneously as possible without the opportunity to give each question much thought.
The Elementary Test
The Psychological Test shined in comparison to this absurd 40-question exam on what was supposedly intended to be a test of general knowledge. Most of the questions were no measure of common knowledge at all, but of trivial facts about China that maybe a few highly experienced veterans of China might have picked up along the way, e.g., for those who have been living in China for at least three years: Do you know the precise month that every Chinese holiday is celebrated in (including all the minor ones that are not considered national holidays)? Why any prospective foreign teacher should be reasonably expected to know the answers to these types of questions, as well as what I counted as 34 other nonsensically trivial questions, is a complete mystery to me. Six questions, of the 40 total, could be argued to measure the type of general knowledge that a well-educated Westerner would or should know (and one of those questions is asked twice—so it’s only five unique questions in total that are valid). At least one of the questions contained no correct answer due to an apparent typo.
Writing Test
This is the only one of the three tests that possesses any face validity as it is based on the types of questions typically found on the writing part of English tests for non-native speakers (e.g., TOEFL, IELTS), i.e., it asks the examinee to write a 200-word essay (about two short paragraphs) expressing an opinion on a common topic in which two points of view are proffered. The user has 10 minutes with which to formulate his position and finish the essay. As my writing sample has not—as of yet—been scored, I can’t address how subjective or not the scoring might be. My approach was to give validity to each side of the debate (although, in this particular case, that was an accurate reflection of how I felt). I am curious to see how I scored on the writing test.
“The Plumber”
As I sat through and then later pondered this “Evaluation System for Foreign Language Expert,” I was reminded of the time a few years ago when the building management—in response to numerous complaints I had made everyday over a two-week period regarding the rather serious plumbing problem in my apartment—sent a “plumber” who was equipped with a Phillips screwdriver in one hand, and a roll of duct tape and a pack of cigarettes in the other. Obviously, he wasn’t the least bit prepared to solve the problem he had been assigned to address—assuming he even had the knowledge.
I commend the SAFEA on its explicit reason for creating such an evaluation system, namely “…to regulate the issuance of permits for foreign experts and to resolve current problems on introducing oral English teachers into the nation’s public education services.” Something like this is long overdue: Unfortunately, this is not the proper way to achieve those goals, not by a long shot.
Pre-employment screening has its place in the world: It is intended to establish a “goodness of fit” between the needs of any given job and the temperament and ability of the various candidates who are seeking the position. Predetermining a good fit between the needs of the job and the capabilities and strengths of an applicant has many benefits, such as a significant reduction in expenses associated with training costs and employee recidivism.
My question to the SAFEA is this: If you are serious about achieving your explicit goals, why not establish a licensing agreement with one of the dozens of time-tested batteries of pre-employment tests that are currently in use? Virtually anyone of the dozens of pre-employment tests, aptitude or vocational interest tests, such as the Myers-Briggs Type Indicator (MMTI) combined with the Strong Vocational Interest Inventory (SVII), would make for a much more valid and reliable pre-employment screening measure. In addition, and completely aside from the absurdity of the instruments you are using, developing a system of administration that can so easily be defeated suggests to me that this is something that was “thrown up at the last minute” to meet a requirement in which there is absolutely no genuine interest in achieving the adopted law’s intent. There are several reputable online computerized testing systems with branches all over the Western world through which your pre-employment battery of tests could be administered to prospective foreign teachers, such as Prometric to name but one. The fact that Microsoft, for example, exercises more integrity and caution in how they certify their systems engineers than China does in certifying its foreign English teachers should be a source of considerable internal concern and reflection.
I would also suggest that there is a much simpler and straightforward manner in which you can achieve your explicit goals: Simply have each province and municipality adopt and enforce the SAFEA guidelines regarding minimum requirements that have already been in place for years. There are numerous Internet services available for checking the academic credentials, as well as the work history and criminal backgrounds of job applicants from all over the world. All you need is an international credit card. The Bank of China issues one as does the Merchant Bank of China, among a few others.
In the end, does it really matter if a foreigner knows on what specific day Chinese Valentine’s Day is celebrated? Does anyone in the SAFEA truly believe that knowledge of such a trivial fact predicts who will be successful as a foreign oral English teacher in China?
This time around, let’s send in real plumbers to fix the plumbing.
Notes- This is an exact replica of the text contained under the menu item “About this System.” [↩]
4 Comments
What Others Are Saying About MKL
...So far, not much has been written about how cultural factors affect what people do online, and, also, how those factors might interact with the online disinhibition effect. Your article is an important step in that direction. Thanks for sharing it with me..., John Suler, PhD, Rider University, The Psychology of Cyberspace.
New Poll - Oct. 5, 2008
Generally speaking, the overall quality of my life is better in China than it was back home.
- Strongly Agree (18%, 11 Votes)
- Agree (38%, 23 Votes)
- Neither Agree nor Disagree (13%, 8 Votes)
- Disagree (13%, 8 Votes)
- Strongly Disagree (18%, 11 Votes)
Total Voters: 61
Loading ...
I simply wanted to write to raise a point about this new test/evaluation. In today’s world of identity theft and easily compromised confidentiality of personal information, I notice that the registration page, and presumably subsequent screens, of this test/evaluation web site are not secured, (secured/encrypted=https, non-secured/unencrypted=http). It is a common, non-secure, unencrypted web site, (http), which I do NOT care for.
However, as I have learned in Life, and particularly in China, we must pick our battles carefully, (don’t fret the small stuff), so I should be tactfully diplomatic in raising this concern with my current/new employer who just a couple weeks ago asked me to take this evaluation. As I’m sure you all know, in China, as opposed to the U.S., often asking questions that begin with “Why…”, or ‘making waves’, are seen as being ‘disharmonious’, and thus undesirably troublesome. In any event, my impression is that this evaluation is a newly enacted requirement, so I wonder if you have had any other feedback raising concerns about the non-secure nature of the web site evaluation.
I’ve also learned that here in China it’s often wise to simply ‘let sleeping dogs lie’, i.e., don’t raise a problem/issue until it actually becomes a problem/issue, sometimes things will simply ‘go away’ over time and not have to be addressed, so I’ll wait to see if the Foreign Affairs Office actually will press me to take the evaluation before I raise my confidentiality concerns with them. As far as I know, a couple other foreign teachers I’ve spoken to haven’t taken the evaluation yet and have expressed some mild sympathy/understanding about my particular concern. So we’ll see how things develop. In the meantime, any thoughts or advice on your part are appreciated.
You raise a very interesting point about security.
Admittedly, I don’t know very much about identity theft. I would think (but I don’t know for sure) that a potential thief would need more than just a name, current address and the knowledge that this person is interested in teaching English in China. At the very least he would need one’s Social Security Number, date of birth, passport number, birth certificate, driver’s license, or some other form of identification that is used to positively establish identity. If all that were required were simply knowing someone’s name, alma mater, and current whereabouts, then anyone who has ever been cited on the Internet would be vulnerable to attack simply by conducting a Google search on his or her name.
I think the fact that the website is not secure is not so much a realistic concern in terms of identity theft, but yet another reflection of the points I made in my article: namely, the whole manner in which this “evaluation system” has been approached is haphazard and invalid. It’s as if they had to do something to be in compliance, so they just threw something together at the last minute without much thought, concern, or care.
My real concern is that the instruments themselves are a complete farce and that these could be used to deny employment to someone who is otherwise quite worthy.
I consider myself to be a relatively well-educated person, with a large fund of general knowledge, but I only knew the answers to maybe seven out of the 40 questions on the “Elementary Test.” In at least one instance, there was no correct answer due to an apparent typo. Most of the questions are just too trivial to be considered a measure of “elementary knowledge,” or are testing awareness of Chinese culture, but in a manner that only those who have lived here for quite some time might be familiar with. Could we be denied employment in the future because of these absurd test results? As I see it, this is the main issue that should be causing all of us a great deal of concern.
Personally, I would refuse to take the exams on the basis that they are not standardized measures with a precedence of use.
I hope this helps.
Dr. Greg,
I read your response, thank you, that you posted on your web site. I’d like to to clarify a couple of things please. You wrote, “As I see it, this is the main issue that should be causing all of us a great deal of concern.”, “this” being the potential of denial of employment in the future because of these absurd test results.
I couldn’t agree more that this is the main concern. As I wrote in my first message directly to you, which you didn’t include in the public posting, “I read your ‘About Us’ screen as well as your recent post about the evaluation itself and I do agree, generally, with your remarks.”
Maybe I’m splitting hairs, but from your response I get the slight impression that, perhaps, you felt I was suggesting that the unsecured nature of the web site was as, or more, important of a concern than the evaluation itself. Not at all, again, I fully agree with you, and didn’t intend to take any issue with what you had written about the evaluation. I only intended to address something that I first noticed when I began to look at the evaluation for the first time, and that you incidentally didn’t happen to mention in your post. I only sought to add an additional concern.
Since I hadn’t gotten as far as the evaluation itself I didn’t know about the nature of the questions themselves, but I had a fair inkling of what they would be like, which your post only served to confirm.
I got as far as the registration screen, which asks for personal, confidential information that I balked at providing on an unsecured web site. (Also, I noticed button options for ‘Next’ and ‘Cancle’ (sic), which along with numerous other glaring grammatical errors, served to only reinforce my fears about what was ahead. Test of English teaching competency indeed! I think SAFEA should ‘Cancle’ this evaluation and go back to the drawing board!)
I think I could/should have worded my initial message to you more clearly. It wasn’t actual, formal identity theft that I was so concerned with, but rather it was the overall compromise of my confidentiality that I didn’t care for. From my pre-education field corporate work in banking, I learned that there are, generally, two critical pieces of information, (other than a name, of course), that are needed for identity theft, at least in terms of committing financial fraud related crimes, and those are date of birth and Social Security number. The registration doesn’t ask for a Social Security number, but it does ask for a good deal of other information that would/should be normally considered confidential, and thus requested via a secured web site.
One other thing I didn’t mention is I’d like to know, (& I suspect you and other readers would as well), is who/what is ‘Chinajob.com’?!?! My impression is that they are some 3rd party placement recruiter who is administrating the evaluation into whose database all of our information will go for who knows how long and for what sorts of undisclosed purposes. Are they an independent 3rd party, or some quasi-part of SAFEA?! And then there’s the evaluation component of Orwellian-sounding ‘resume filtration’?!, which appears to be voluntary, I hope.
And, of course, all this about information confidentiality is subordinate to the paramount problem of the nature of the evaluation itself in terms of serving as a possible criteria for employment consideration, which you have already written about in your initial post.
(You write that in consideration of copyright concerns, you aren’t posting the evaluation questions. I’m sorry, but since when has China been concerned with respecting copyright laws?! Oh, only when its THEIR copyrights that are at risk! Maybe you might go ahead and post the questions and when/if you get some complaint, refer them to the corner DVD shop, street Rolex vendor, etc.! ha!)
Yes, the idea of a pre-employment screening filter/evaluation has its merits, but the way they’ve got this set up now will mean, if used as a criteria to grant employment, the throwing out of a lot of babies with the bathwater…bathwater being the wholly unqualified as teachers unwashed ‘backpackers’.
(Like it or not, we are citizen ambassadors here in someone else’s country. And, at least I feel, I should conduct myself as such. I am a guest in these people’s country; this is something I try to keep in mind when I walk out my door each day. My parents, of the WW II generation, taught me to be polite when in someone’s home. You don’t go to someone’s house looking/acting like an ill-mannered slob. And, regardless of how my ‘hosts’ act at the local stop when the bus pulls up, I try, often with frustratingly little success, to politely accommodate the ladies and elderly folks while others are trying to elbow me aside….so why don’t I just take a cab? Because the bus is more of an adventure!…and isn’t that what Life is all about? ‘Either get busy livin’ or get busy dyin’! ‘Don’t just eat a hamburger, eat the hell out of a hamburger!’)
Thanks, Concerned Foreign Teacher
Concerned FT,
The entire English language program in China is a complete mystery to me. How a country that possesses some of the finest minds in the world could come up with a foreign language program that is such a dismal failure is mind boggling. I am almost beginning to believe that, in reality, the Ministry of Education and SAFEA do NOT want Chinese nationals to learn English despite the fact that they are required to offer it as a legitimate curriculum.
The Chinese English teachers teach English almost entirely in Chinese and then believe that exposing students to a native speaker for 90 minutes per week can offset the futility of that approach? There is no real attempt at providing an English language environment in any university English language department that I’ve seen or even in private English language schools (although I suspect the ubiquitous and pedagogically worthless “English Corner” is supposed to fill that void). Yes, I can use chopsticks. Yes, I like Chinese food… really.
If one of the main goals of this evaluation system is to eliminate alcoholics and those who have problems with explosive tempers, there is a much simpler and reliable way to do that: Order a pre-employment background check over the Internet. This typically includes a criminal background check, verification of work history and validation of university degrees. If there is a history of alcohol abuse or intermittent explosive disorder, surely these will come to surface in that report.
Instead, they create an “elementary test” with such questions as:
“20×125x5×4x8=250=?”
Now, in fact, the ability to perform simple mathematical calculations is associated with a clear mental status. However, these types of questions are supposed to be administered without access to a calculator! If someone is taking the exam from their computer, I have to believe they will simply access their calculator accessory and plug in the numbers. So, in that context, deriving the wrong answer is not an indication of mental difficulty but, perhaps, of terrible fine motor skills or maybe legal blindness.
Here’s another example:
“What color are majority of Chinese people’s eye?”
Now here is the interesting thing about this question. Of course, there is no such thing as black eyes (on a human being, barring the presence of an eye disease called Aniridia) but every Chinese person I have ever known has told me that his or her eyes are black. I once informed my class that there is no such thing as black eyes in healthy human beings and they corrected me, admonishing that all Chinese people have black eyes—they know this because their Chinese science teachers told them so in biology class. I am willing to bet the house that the “correct” answer to this question is black (particularly as the correct answer would actually be “dark brown” as opposed to brown).
Getting back to the question of the evaluation system, if the Chinese government truly wants the Chinese to be able to communicate in English, they will have to create a genuine need for it completely aside from just passing school admission tests. The government will have to adopt it as an official second language and core courses starting from primary school will have to be taught exclusively in English. This is basically the model they use in the Philippines and guess what? Pretty much everyone there can communicate at least somewhat in English. English, as an academic subject, is taught only by Filipinos and despite the fact that they don’t use foreign oral English teachers, even common people with relatively poor education can communicate in basic English.
Until China eventually creates a true English language environment and a real need to communicate in English, I don’t think it matters much whether their foreign English teachers are former award-winning professors of linguistics or “backpackers” who are just traveling through, no matter how emotionally well-adjusted each may or may not be.