You beat the deadline and got your Federal performance reports submitted by December 31.  Congratulations!  Now, as you’re reviewing the data, are certain questions popping up in your head?

·        Did all of your local programs use the same criteria to determine completion of educational functioning levels? 

·      Were the methods they used valid and reliable?

·        What about those students who were not post-tested?

·        What about assessment documentation for work-based project learners?

·        Are teachers complaining that there’s just too much paperwork anymore?

·        Did your whiz bang data collection system lose some fizzle when you put it to the test?

Well, ponder no longer!  Make a resolution that 2002 will be the year that you’ll tackle those issues head-on!  Thanks to financial support from DAEL, a variety of capacity-building opportunities will be available for you and/or your state staff in 2002.  NAEPDC will be designing resources aimed at enhancing your accountability system and strengthening your program improvement capacity.

We know that your schedule is very demanding, so we want to make sure that these resources meet your particular needs.  To do that, we need input from YOU!  Attached is a brief self-assessment called Decision Points.  The purpose of Decision Points is to (1) give you an opportunity to develop a “profile” of your current practice based on key decision points and (2) give NAEPDC and other contractors an idea of what resources would be most beneficial to you.

Please take a few minutes to complete the self-assessment and submit electronically by January 30.  Remember:  as you go through the questionnaire, resist the urge to be overwhelmed.  You can’t redesign your system overnight.  Choose one area in 2002 that you really want to improve, and go for it!  NAEPDC will be there to help.

During the February DAEL meeting, we’ll give you an overview of all the resources that await you!  In addition, if you return Decision Points by the January 30 deadline, your name will be put in a drawing for a $100 gift certificate to L.L. Bean, redeemable at any of its retail stores or on-line catalog.  The drawing will occur at the February meeting.  Good luck!

 

As the following self-assessment will attest, you, as a state director of adult education, have a very complex job!   You are expected to create performance and accountability systems that ensure continuous improvement in program services, in addition to the myriad of other responsibilities.  We know that this is not an easy job!  NAEPDC wants to help!  

Decision Points is intended to help you examine some of the important issues that must be addressed in enhancing your accountability system.  It is certainly not an all-inclusive list, but it will provide a starting point – a roadmap – to help you determine where you need to go.  Decision Points addresses three main functions:  assessment, data collection, and use of data for program improvement.  For assessment, you’ll explore the policy, curriculum, and training decision points that must be addressed in developing a sound, research-based system.  For data collection, you’ll look at key questions related to management information systems, policies and procedures, fiscal resources, student follow-up, and training that are inherent in a thorough and efficient data collection system.  And finally, you’ll examine the policy issues, training needs, and fiscal resources associated with the use of data for program improvement.

DON’T PANIC!  Few if any states have addressed all of the items listed in Decision Points.  By prioritizing and approaching your accountability system strategically, though, you can begin to build a continuous process of program improvement.  You will probably find that you are already doing many of the items listed in the self-assessment.  That’s great!  Perhaps, however, you would like to learn about new or better ways of doing them.  Perhaps there are some items that you are not doing now but, given your present situation, are not able to make them a priority at the present time.  That’s OK, too.

The important thing is to decide what you are doing now, what needs to be done (now or sometime in the future), and what items are priorities for which you would like assistance or training.  So sit back, relax, and take a few minutes to complete Decision Points.

Listed below is a series of key decision points to guide you through the review of your accountability system.  (1) Select “yes” or “no”  under  the “Doing Now” column for each numbered item that you presently have in place.  (2) Then select “yes” or “no” under “Want Help” for any item for which you would like assistance even though you may presently have the item in place. Answer the final three questions and hit "submit."

(Please complete)

Name of State Director:            State:

I.  Assessment

A.  Policy Issues:  Do you have a written policy or document that:

 Am I Doing This Now?

Do I Want Help?

1.   Requires that one (or more) specified assessment instruments be used for reporting the Educational Functioning Levels (EFLs)? 

2.   Utilizes assessment instruments that are specified on the NRS guidelines?

3.   Provides guidance on the use of standardized alternative assessment procedures  that are aligned to the NRS descriptors for placement and/or completion of EFLs? 

4.   Provides guidance on selecting appropriate assessment instruments for sub-populations of learners (e.g., ABE, ESL) and for matching pre-and post-tests by parallel form and level? 

5.   Specifies a time period (calendar dates or minimum hours of instruction) for administering pre- and post-tests? 

6.      Specifies guidelines for determining which learners should be classified as “work-based project learners”? 

7.      Specifies the assessment process to be used for work-based project learners (standardized pre/post tests and/or performance based assessment w/standardized rubrics)?

8.   Specifies the state-approved test and passing standards for documenting achievement of skills to pass the citizenship exam?

B.  Curriculum Alignment:  Does your curriculum:

 Am I Doing This Now?

Do I Want Help?

1.   Utilize state-adopted content standards that are aligned to the EFLs?

2.   Specifically match the content of the assessment instruments being used?

C.  Training and Technical Assistance Issues:  Do you have a training and technical assistance system in place that:

 Am I Doing This Now?

Do I Want Help?

1.   Includes a written document or manual that provides instruction about the procedures for the administration of assessments?

2.   Provides guidance on appropriate placement into EFLs based on NRS benchmarks, including guidance for using standardized tests that are not presently aligned to NRS benchmarks (if applicable)?

3.      Has a training program in test administration and assessment procedures that includes participation requirements (voluntary or mandatory), frequency of training, and guidance on your state assessment policy listed in Part A above?

4.      Includes training in assessment monitoring for local program administrators to ensure local staff are following appropriate test procedures? 

5.      Includes a person (yourself, a staff member, vendor) who provides technical assistance to local programs regarding administration and submission of assessment data?

II.  Data Collection

A.  Management Information System:  Do you have a system that:

 Am I Doing This Now?

 Do I Want Help?

1.      Utilizes a state-adopted computerized student record system to enter NRS data?

2.   Has a relational database whereby information on individual students can be related to other variables in the database?

3.   Has the ability to aggregate data at the state to produce required Federal reporting tables?

4.   Has the ability to produce state and local reports to be used by state staff, program managers, and teachers to improve program quality?

5.    Has edit checks for test scores, appropriate test forms, appropriate pre-post test time lapse, and other data integrity?

B.  Data Collection Procedures and Policy:  Do you have written guidance on:

 Am I Doing This Now? Do I Want Help?

1.   Data reporting timelines for the state (quarterly or more frequent)?

2.   The roles and responsibilities of local programs in data collection and input?

3.   NRS definitions and procedures for determining and submitting student demographic and daily attendance data?

4.   Determining appropriate student goal setting and completion of goals?

5.   A system for monitoring quality control and data integrity in local programs (e.g., random review, on-site data audits)?

C.  Fiscal Resources:  Do you have a process for:

 Am I Doing This Now? Do I Want Help?

1.      Ensuring sufficient local funds to collect and input the data through extra designated funds in local budgets or allocation from existing budgets?

D.  Student Follow-Up:  Have you:

 Am I Doing This Now?

Do I Want Help?

1.      Determined the most appropriate, accurate, and realistic follow-up process to use in your state (data match and/or student survey)?

2.      Implemented a process to safeguard student confidentiality issues?

3.      (If using data match) Entered into interagency agreements for conducting a data match on employment data? 

4.      (If using data match) Entered into interagency agreements for      conducting a data match on postsecondary education/job training?  

5.  (If using student follow-up) Specified the survey methodology and protocols that local programs must follow if implementing local survey follow-up?

E.  Training and Technical Assistance:  Do you have a system in place that:

Am I Doing This Now?

Do I Want Help?

1.   Has a training program that includes participation requirements (voluntary or mandatory), frequency of training, and guidance on your data collection policy and procedures?

2.   Provides training on accurate data entry?

3.   Includes training in data collection monitoring for local program administrators to ensure local staff are following appropriate procedures? 

4.   Includes someone (yourself, a staff member, vendor) who provides technical assistance to local programs on issues related to the use and upkeep of the program’s data system?

III.  Using Data for Program Improvement
A.  Policy Issues:  Do you provide guidance on:  

Am I Doing This Now?

Do I Want Help?

1.      Supporting local programs in developing a formal process for     analyzing and using data (e.g. program improvement teams)?  
2.      Whether the analysis/use of data process is mandatory as a pre-requisite to funding or voluntary by local programs?  
B.  Training and Technical Assistance:  Does your training design include:  

Am I Doing This Now?

Do I Want Help?

1.   Determining appropriate issues to analyze?

2.   Determining what data sources to use that address the issue?

3.   Using frequency distributions or other statistical procedures to analyze data?

4.   Using relational databases or statistical process to compare and contrast data?

5.   Analyzing the data and forming conclusions about what the data says?

6.   Determining a program improvement plan based on what you have learned?

7.   Training for both teachers and local program administrators to ensure local staff are analyzing and using data to guide and manage program improvement? 

8.   Someone (yourself, a staff member, vendor) who provides technical assistance to local programs on issues related to the analysis and use of data for program improvement?

C.  Fiscal Resources:  Do you provide guidance and support for program administrators to:  

Am I Doing This Now?

Do I Want Help?

1.   Adjust program schedules to provide time for teachers to analyze data, seek and experiment with alternatives, and make program changes?

2.   Direct fiscal resources to support teachers using data to improve instructional services?

         

1.   I, or a member of my staff, would be very interested in learning more about:

(check any that apply)

   Assessment

    Data Collection

    Using Data For Program Improvement  

2.   I would prefer receiving this information through:  

a capacity-building  workshop                  

NAEPDC website

individual consultation with NAEPDC staff or topical expert

3.   My preferred location for a capacity-building workshop is:

     Eastern United States

    Western United States

    Southern United States

    Northern United States

    No preference, just someplace warm

     No preference