The Evolution of Profiling
at Self Management Group
An Historical Background Note
The Self Management Group (SMG) consists of several companies engaged as management consultants with a focus on helping organizations select and develop people who will survive and perform effectively in a variety of business (and other) environments. The Self Management Group includes the Self Management Resources Corporation whose primary focus for over two decades has been developing and consulting on selection and retention systems for major clients throughout North America and increasingly in Europe and the Far East .
A major factor in the success of the Self Management Group has been its development and ongoing research into the factors that make people successful and the measurement of these factors using a variety of proprietary normative profiles.
The initial SMG profile was the Personal Orientation Profile (POP) which was an innovation in normative profiling used in selection (a comparison of normative and ipsative profiling is included in this document).
Since its early development, the POP has become the instrument of choice for many large, successful competitive sales forces, particularly in the financial services sector as well as others such as technology sales, automotive sales and other demanding environments.
The POP has become the popular tool that it has because it is predictive and it creates normative data which allows for the comparison of the personality traits and attitudes of different individuals. These factors have been shown to be related to success in these demanding roles.
The POP has been thoroughly validated on a regular basis for the last two decades and more. An important fact about these validations is that they are both industry and client specific and become strategic tools to assist SMG's clients with improved performance and retention. Our validation studies become the intellectual property of our clients and the data is not shared with other clients. In some instances, cross-sectional data from diverse clients from different industries and business culture are used to provide generic norms for an SMG profile. These papers are also available from the Self Management Group.
The POP was originally copyrighted in 1978, revised in 1979 and on several other occasions, most recently in 2001. It has been translated into French, Spanish, Chinese, Japanese, Greek and several other languages.
The POP also became the basis for all of the other profiles which were subsequently developed by SMG. These include but are not limited to:
ManagementPro (MPP3) for selection and development of managers;
ServicePro (QSP) for selection and development of customer service with emphasis on call centers;
ProfessionalPro for selection and development of professionals in business and consultant roles;
SelfEmplomentPro for self employment development and planning;
SportsPro for athletes and coaches in competitive sports;
and more to come.
All of these tools are normative and are based on the research that has been ongoing ever since the initial research in 1977 and 1978. All of these profiles have been used as coaching and development tools and most have been used in selection, retention, performance management, team building, matching to managers, succession planning and general career counselling.
In addition to profiling, the Self Management Group consults on other strategic human capital issues such as recruiting, screening, selection and career development, behavioral based interviewing and career self management.
The Self Management Group also provides unique training programs based on the premise that self managers and self managing work cultures produce the best performance and the most personally gratifying work experiences. Training that focuses on managing effort, self management, team building, stress and lifestyle management and specific customized programs are all available and use the SMG profiling as aids to development.
The remainder of this document provides some of the background of profiling at the Self Management Group. It is presented in the following sections:
I The Personal Orientation Profile
The Personal Orientation Profile (the POP) is a powerful, state-of-the-art psychometric instrument which has been used to profile more than a million people since its original development in the late 1970's at the Self Management Resources Corporation in Toronto , Canada . The POP assesses and reports on the behaviors of individuals in highly competitive, demanding roles such as competitive sales
Purpose and Theory
The POP was designed as the psychometric instrument as a component of a broader system to assess persons in high demand work environments.
The original theory behind the test construction was that high demand work environments require powerful personality characteristics for someone to survive and perform effectively. The further hypothesis was that these characteristics could be assessed reliably and that all individuals possess these characteristics to varying degrees.
Based on these assumptions, the ultimate goal of the POP was to predict performance and become a valid selection tool in hiring people for high demand work environments. The strategy chosen to achieve this goal was to imbed the POP into a total system which included testing and performance assessment which could be used for continuous research, updating, validation and re-validation until the desired objectives were reached.
To achieve the desired goals, it was necessary to develop a system which used the POP to create a dynamically evolving data base which could take advantage of the latest micro-computer applications as they were developed.
Test Construction and Interpretation
Primary Traits Represented in the Profile
One of the features of the development of the POP was the identification and definition of the desired characteristics in advance. This differed from the more typical method of leaving item selection to the differences found between criterion groups. Each scale was conceptualized by identifying factors associated with high demand work environments. The psychological literature associated three major factors with success in these environments: competitiveness; achievement orientation; and independence. Each of these three factors were developed as bi-polar scales which represented a continuum that would show a very high presence of the factor at one end and an almost total absence of the factor at the other end. These bi-polar scales consisted of a Power trait and a Neutralizing trait, each of which indicated the strength of the presence or absence of the trait. By combining them, the strength of measure was enhanced. These became the basis for the original item pool which was subsequently augmented by the need to evaluate sociability and approach to technical details.
The POP was constructed to include the 8 trait scales which were selected from the psychological literature available. A major source the initial item pools was Allport and Odberts' (1936) dictionary of 4500 trait items. Each item underwent a critical screening by two editors to ensure its conceptual link to the characteristic being measured.
The 8 individual scales measure character traits which are found in everyone to a greater or lesser degree. They were divided into 2 groups labeled as Power and Neutralizing scores. The Power scores are those which are considered as strengths in a competitive sales environment. These are paired with Neutralizing scores which tend to moderate the corresponding power scores. The two groups of 8 traits are as follows:
Power Scores Neutralizing Scores
Enterprising (Ent) Acquiescent (Acq)
People Oriented( PO ) Investigative (Inv)
Achievement Orientation (Ach) Relaxed (Rel)
Independent ( Ind ) Team Oriented (Team)
Each of these traits is measured on a 21-item scale of correlated adjectives and brief phrases. Each item is scaled on a 5-point Lickert scale which allows every item and every scale to be scored in normative manner. Because of the number of items in each scale, there is considerable psychometric depth and the items have low face validity making it extremely difficult for anyone to fake their answers in a manner that would enhance their desirability as a candidate.
The original list of 300 items was reduced through factor analyses to 168 items based on the production of 4 bi-polar clusters that assessed competitiveness, achievement orientation, independence and sociability. Biserial correlations were computed for each item and the scale to which it belonged. For inclusion each item had to display a higher correlation with its own scale than any other scale. Items retained also required an endorsement of between .05 and .95. It was found that items with extreme endorsements usually carry little information, have unstable correlations, tend to be high on social desirability and usually possess modest reliability and validity.
The list of items derived from the preliminary work was formed into the earliest version of the POP which was administered to a sample of 198 adults over the age of 19 and employed. Included in the sample were 87 persons who were employed in competitive sales environments in real estate, life insurance and automotive sectors. For the group of sales people, performance assessments in terms of net earnings were obtained as criterion measures.
To facilitate interpretation of each scale, a step-wise multiple regression analysis was performed on each scale using the scale score as the dependent variable and the items as the independent measures. The primary scales were also subjected to principal axes factor analysis using R2 values as communality estimates and a quartimax rotational solution.
Secondary Traits in the POP
As the POP analyses continued, 3 additional scales were added to evaluate locus of control, somatic complaints and social desirability. These provided invaluable insights to the interviewer who used the POP in the selection process.
Reliability of POP Scales
The individual POP scales were split into odd and even numbered items and each half scale was correlated with the other half. The correlations were adjusted using the Spearman-Brown formula. The adjusted correlations were acceptable and ranged from .83 to .92.
Test/Re-test Reliability
The test/re-test reliability was established by randomly selecting 25 subjects (approx. 12%) and having them re-do the profile. A period of one month separated re-testing from the initial testing. The scale scores from the 2 testing's were correlated and ranged from .90 to .97.
Internal Consistency of the POP
Alpha reliabilities were recently calculated to provide evidence of the internal consistency of the POP scales. The POP character trait scales demonstrate good internal consistency with alpha reliabilities ranging from 0.76 for the Acquiescent scale to 0.93 for the Enterprising scale.
Validity of the POP
The POP was designed to predict performance in competitive environments and as a result, the original validation of the POP focused primarily on concurrent validation studies. Multiple regression and discriminant function analyses were the primary statistical procedures used in the research.
Original Study (1978)
In the original study, the POP constructs were analyzed in relationship to the net income criterion which was available for the 87 competitive sales people in the initial sample.
In summary, the analyses showed that the initial theory about succeeding in a competitive sales environment had been correct. There were linear relationships between performance groups based on the factors competitiveness, achievement orientation and independence and to a lesser degree sociability.
It was also important to note that the bi-polar scales were even better predictors of success than individual scales. As the differences between the Power and Neutralizing scores were narrowed, the performance levels dropped. Individual scales were shown to identify differences between different levels of performance as well.
Cross Validation (1980)
The follow-up cross validation in 1980 was performed on 214 sales people with performance data available and suitable minimum experience.
Using discriminant analysis techniques, the POP was able to predict group membership in the correct group out of 4 groups in 60% of the cases. Only 14 of 214 (6.5%) would have been selected by the POP and not achieved at least an average level of performance. This very low level of false positives provides a very strong reason for feeling that a strong POP prediction will lead to satisfactory to superior performance in the vast majority of cases.
NB. The interested reader may obtain detailed information about the statistical procedures and analyses from the Self Management Group.
Evolution of the POP
As more and more competitive sales organizations began to use the POP in their selection models, the POP was validated over and over again in a wide variety of sales cultures ranging from financial services such as life insurance to automotive sales and real estate sales etc. Each validation study allowed each organization to understand its sales force better and to more accurately identify the characteristics of sales people at different performance levels. From this, customized formulae were derived for each client with sufficient POP and performance data.
In addition, each validation study helped the Self Management Group learn more about the unique characteristics of selling in different vertical and horizontal markets as well as learning which personality styles will work more effectively in different regions. Even differences between rural and urban selling styles have been identified and characterized.
The exigencies of the marketplace and legislative change also affected the POP. As more data became available, there were minor re-calibrations of the POP. Client demand led to the integration of a scale used to measure the individual's approach to call reluctance which is a significant attitudinal factor which can be a performance block in sales people.
The Americans with Disabilities Act led to the changing of the Somatic Complaints scale to a version that focuses on integrating a demanding career with a busy lifestyle. EEOC guidelines mandated additional reviews to assure that the POP and its off shoots did not have adverse impact on minority groups.
Client demands also led to the use of the POP psychometric engine as the basis for instruments that were more career and position related. Our research showed that as well as predicting success in demanding environments, the POP primary traits would also assist clients with fitting individuals into different career paths, more administrative or process-oriented roles.
Some of the more widely used POP based psychometric tools include the SalesPro, the ManagementPro, the CareerManagementPro and the ProfessionalPro.
In addition, SMG researchers developed a second psychometric engine which is used for the ServicePro, the ITPro and the SportsPro.
Each of these profiles follow the same methods used in developing the POP and are validated as required.
II Profiling and Selection
It is important to recognize the difference between screening and selection. A good selection system begins with a good recruiting system which delivers suitable candidates as needed. Many competitive sales organizations recruit continuously to keep an on-going flow of candidates for those sales roles with low retention rates.
Ideal selection is based on large numbers of suitable recruits who are systematically screened until only viable candidates remain. Modern web-based screening makes reviewing non-negotiable requirements and evaluating recruits based on work history, education and other demographics a very cost effective solution that allows HR professionals to focus on more rewarding tasks. Electronic screening allows for recruits to be rated and the quantity and quality of flow adjusted with ease. Those who pass electronic screens are then reviewed and may be further screened by a telephone interview until they actually enter the selection process.
The selection of personnel is one of the most important things an organization does. Virtually all organizations interview candidates and, as most HR professionals know, a structured, competency interview by a capable interviewer is an extremely important part of selection. Most successful organizations also recognize the need for profiling as part of their selection process to provide an objective look at each candidate.
The Self Management Group has been recommending a three-part process to its clients for many years. We recommend profiling combined with both structured and unstructured interviewing to get the most rounded, complete assessment of a candidate at a reasonable cost per selection. By putting selection into a systematic process, the organization is able to minimize personal bias and capture data that will allow the system to be a model for continuous improvement.
The data from a normative profile such as the POP or other SMG profiles is captured automatically into a database for each client. The value of a good structured interview is that it assesses the competencies that are important in the role for which the candidate has been recruited. Using behavioral based questions and asking the same ones to each candidate brings a systematic approach that can be measured. If the interview is consistent, it can be conducted by different interviewers and get consistent results. Finally, the unstructured interview helps the interviewer establish chemistry, how well the candidate will fit the existing team, get a feel for potential growth and return on investment. If all three components are used, the selection system will become predictive and a powerful strategic tool.
The Selection Rater
The most effective way to implement the three-part selection strategy is to use what the Self Management Group calls a selection rater. This allows the organization to integrate the three components which complement each other into a single document.
The three components are:
I SCIENCE (Objective Profile)
II EFFORT HISTORY (Structured Interview)
III FIT (Organizational Fit)
A well rounded selection process will contain all of these factors to make the best possible selection decision.
I SCIENCE
The SCIENCE portion comes from the creation of position related norms using the appropriate SMG normative profile customized to the position and culture.
II EFFORT (Structured Interview)
The EFFORT component is based on a structured interview that focuses on the competencies identified for the specific position.
III FIT (Unstructured Interview)
The FIT is a subjective view of how well the candidate will represent the organization, interact with existing team and match to the manager. 
Each of these three components is sources of data which becomes the basis for on-going validation and continuous improvement. The data becomes information which is transformed into knowledge about the culture and the selection system. This in turn becomes a powerful source of HR strategy.
III Profiling and Development
The POP was originally designed as a selection tool for competitive sales people but as it evolved, managers, career coaches and others found that it provided valuable insights into how to manage people as well has how to help them develop their careers.
Focusing on the POP as a coaching tool for sales people, it provides many insights into the individual's normal behavior based on their traits and attitudes and offers suggestions on how to deal with them.
The ability to identify issues like call reluctance or reluctance to close provide managers with opportunities to focus their coaching efforts in a way that will maximize their own efforts and enhance the sales representative's success.
Similarly approach to structure, degree of self management, communications style are all clearly identified and will allow astute sales managers the opportunity to coach individually for maximum results.
Use of the POP and its derivatives for more than 20 years has taught us a lot about the use of these tools for both coaching and for personal development of one's potential and career.
The SMG tools have been used extensively for individual coaching as well as team building, career development and succession planning. As these topics are quite extensive, we can only mention them here with the recommendation that more information is available through Self Management Group or one of its consultants or distributors.
IV Normative and Ipsative Profiling
There are two main types of psychometric profiling: ipsative and normative. Ipsative profiling is sometimes described as intrapersonal in that it deals with the various attributes of a single individual whereas normative profiling allows the scores of each individual to be placed on scales where norms exist for a population. The POPT for example, has developed norms for sales people so that anyone taking the POP will be compared to other sales people.
Ipsative profiling is easily identified by questions that require forced choices such as True/False or Yes/No. Another form of ipsative profiling will ask the individual to rank order several attributes that are presented (often in groups of 4 items). This forces the individual to make choices with which he or she may not agree.
Ipsative scores often appear to be effective for profiling candidates for job selection, but there are a number of reasons why this method of profiling should be abandoned for selection purposes.
On a conceptual level, these scales force respondents to make a decision between absolute categories (e.g., True/False) when neither category may accurately represent the unique experiences of participants. Ghisselli in the 1940's indicated that this type of format could frustrate respondents and this may then lead them to use response sets (e.g., alternating between True & False regardless of the nature of the question).
Forced choices between discrete categories may be satisfactory in tapping into extreme scores but they are far less effective in making discriminations between mid-range scores which represent the majority of the population.
The major disadvantage of an ipsative scale is the inability to collect normative data, as the scores generated by ipsative scales are rank-ordered and not comparable across individuals. It is impossible, therefore, to validate the effectiveness of these profiles. Hence, by virtue of their psychometric property, ipsative scales are "only useful as a basis of discussion" with the individual candidate who has completed them (Kline, 1993, p. 61).
Normative profiles (e.g., POP) are a more modern psychometric approach and offer a range of response choices (usually 5) that allow respondents to choose between continuous categories that are more representative of their experiences. The data generated by these scales are normative and can be compared across individuals allowing for the effectiveness of these profiles can be validated.
Paul Kline who is regarded by many as one of the world's foremost authorities on test construction, "There seems no reason to use ipsatively scored tests and they are not recommended".
Kline, P. (1993). The Handbook of Psychological Testing. NY: Routledge .
V Frequently asked Questions
What does your instrument test for and how?
The Personal Orientation Profile (POP) was created specifically for the selection of top performing sales professionals. It measures 3 major factors that predict performance: (1) Enterprising Potential (EP) which measures prospecting potential (business development) and predicts the activity levels and self management potential of a candidate; (2) Achievement Potential (AP) which measures motivational mix and predicts the sense of urgency and closing style of the candidate; and (3) Independence Potential (IP) which measures the need for structure and coaching and predicts retention and fit to a specific culture. These 3 factors combine to create the Predictor Score (PS) which predicts overall sales performance.
In addition, the POP assesses the communication and learning styles through the People Orientation (PO) and Investigative (INV) scales. Sales people tend to be high on PO and modest on INV.
Finally, the POP measures the important attitudinal factors of Self Confidence, Lifestyle Management and Call Reluctance which has 4 sub-scales: Sensitivity to Rejection; Prospecting Natural Markets; Commitment to Product; Commitment to a Sales Career.
The combination of suitable character traits and attitudes has been shown to be effective predictors of performance in virtually every study of sales performance that we have conducted.
Other POP based instruments use the same character traits and modified attitude scales and occasionally emotional intelligence measures to assess suitability for a variety of non-sales roles including management.
Is profiling best used for front-end screening, in-depth selection or both?
The POP and its derivatives are selection tools that can also be used as a guide to help train, develop and coach a new or experienced representative. All SMG profiles include candidate feedback that is designed to build self confidence and offer the candidate valuable career management advice to enhance the recruiting process. The management reports also include questions to ask in the selection interviews and offer training and coaching suggestions. All of these are available on line and provide feedback on a 24/7 basis.
SMG has also developed cost effective screening tools (e.g. POPSCREEN) that are available 24/7 in real time. These are suitable for pre-selection screening and substantially reduce the cost per hire and the time to hire. The integration of the POPSCREEN with the recruiting services of SMG can provide an organization with a continuous supply of pre-qualified candidates.
Is it possible to fake your results on the POP? How can we tell?
It is very difficult to fake the POP or other SMG profiles because the section that deals with character traits has low 'face validity' which means that it requires expertise in trait analysis and test construction to understand what factors the profiles are examining. The number of items (21 per trait scale) provides for considerable psychometric depth and as all of the items are strengths in some context, candidates who plan to enhance their profiles artificially find it difficult to do so.
In addition, the POP and derivatives use the Uncertainty scale to assess whether the candidate is answering in a manner which is socially desirable (i.e. giving answers to put selves in best possible light. High scores on the Uncertainty scale provide an indicator that the person may be misrepresenting self.
How do you educate field management in the proper use and interpretation of results for your instrument?
For all new clients we conduct workshops on how to interpret the profile and how to integrate the POP with the other components of the selection process, such as interviewing. We have developed a Selection Rater system that combines the POP with structured interviewing and unstructured interviewing. The workshops can be conducted by an SMG consultant or we can train the trainer.
The SMG website includes an online training module, a PowerPoint presentation and an interpretation manual. Our services also include a free telephone consultation on individual profiles through our 1-800 customer service line which provides for ongoing training and consultation. Finally, we conduct an annual validation study and present the findings to managers and recruiters to help improve future selection decisions.
For an organization that is testing a large number of candidates at the same time, how well do you support paper-based or other mass-testing procedures?
SMG has several methods for administration and scoring which provide the client with maximum flexibility in profiling unlimited numbers of candidates. The POP is available on paper or via the Internet or company intranet. It can be scored on a local PC, on the Internet, or through our scoring center via fax or telephone.
SMG develops a branded, vanity website for all our clients that includes the on-line training materials and a password protected profile administration center that allows the client to manage the profile suite 24/7.
How well does your instrument predict success on the job? Explain
The goal of the POP is to predict both performance and retention. We have conducted hundreds of predictive validation studies with all types of distribution systems. SMG conducts an annual study for each client with the goal to improve the predictive capability of the selection process. This creates a continuous improvement model and provides an organization with ongoing knowledge about the effectiveness of recruiting sources, screening criteria, selection tools and retention strategies.
The SMG research team continually collects E.E.O. data to ensure compliance with all guidelines and legislation. For individual clients, we have included data from the complete recruiting process to ensure compliance of all the components of a process. These studies have demonstrated that the POP is fully compliant and the database adds credibility to a client's system and supports the objectivity of a client's hiring practices.
We have a validation manual, sample studies, executive overviews and marketing materials that highlight the predictive capabilities of the POP. SMG has the largest database (over 2 million) of candidates in the financial services industry. A significant number of these candidates were hired and their performance and retention data were used to validate our predictive models.
The validation study includes a written report that highlights the success factors across the total company, at a regional level and a branch level. This can help tailor the selection criteria to any differences within an organization. The validation results can then be used as the benchmark to compare new recruits and programmed into a customized page in the POP report.
From a statistical perspective, insurance companies that use the POP as part of their selection process have on average double the 4 year retention of the industry average. As a predictor of performance, if the POP is predicting superior performance, survival rates are typically in the 90-95% range. The Predictor Score (PS) on the POP typically displays a linear relationship to actual performance. For example, low performers who had an average PS score of 23 achieved only 75% of the sales objective, average performers who scored 32 on the PS achieved 100% of the sales objective and the high performers who averaged 42 on the PS achieved 150% of sales objective.
In summary, the POP can improve both retention and performance simultaneously. Many selection systems tend to improve performance at the expense of retention or retention at the expense of performance. Through our validation studies, which provides for a continuous improvement model, our clients have realized significant growth in both number of agents and production per agent.
|
