ISQ and Student Success

Questioner: Anonymous

Posed to: Pamela Chally, Intrim Provost and VP of Academic Affairs

Answered by: Pamela Chally, Intrim Provost and VP of Academic Affairs

 

Question 1:

The recent emphasis on ISQ bothers me as an educator. Is it my role to educate the students or make them happy? Please do not respond saying these are the same. If you want references and recent research, ISQ does not determine that. Many universities are on the path to make the ISQ result confidential and available to only the faculty while UNF posts them online by faculty and course number. Questions. 1. To what extent have you made sure that the improvement of ISQ is not coming at the cost of course quality? Many faculty are putting excessive weight on attendance, take home exams, open book exams, and grade curves so that students like them. How are colleges controlling that? Or is it a college’s policy to do only what is necessary to improve the metrics and get state money? If so, can we change the mission statement to low quality instead of high quality education?

We have been clear about our goals for UNF to increase our standing in the performance-based metrics, but never at a cost of a quality education for our students.  To draw attention, in general, to quality of instruction can only benefit students.  I do not have any data indicating that faulty are putting excessive weight on attendance, but attendance in itself is an important factor in students doing well in a class.  Alternative means of assessment are totally the decision of the faculty member.  Consequently, the choice of an open book examination is an individual faculty decision. As Academic Affairs has drawn attention to student success, we have also provided support to both faculty and students.

 

Question 2

It has been proven that ISQ scores are discriminating against immigrants (especially with an accent) and women. Is it in UNF’s new mission to use a statistic that has been proven to be discriminating? If the administration thinks that is not true in the case of UNF, have they done a study to make certain?

In response to a recent discussion with Faculty Association on this topic and suggested language forwarded by Terri Ellis which shows impact on gender and racial bias in evaluations, Academic Affairs requested from ITS the incorporation of suggested language in the ISQ instructions to students:

 “Student evaluations of teaching play an important role in the review of faculty. Your opinions influence the review of instructors that takes place every year. Iowa State University recognizes that student evaluations of teaching are often influenced by students’ unconscious and unintentional biases about the race and gender of the instructor. Women and instructors of color are systematically rated lower in their teaching evaluations than white men, even when there are no actual differences in the instruction or in what students have learned.  As you fill out the course evaluation please keep this in mind and make an effort to resist stereotypes about professors. Focus on your opinions about the content of the course (the assignments, the textbook, the in-class material) and not unrelated matters (the instructor’s appearance).”

 

Question 3

If the administration is so concerned for students, why don’t we drop the full time student requirement from 15 to 12 credits? Most students at UNF are employed, often even more than 40 hours. So forcing them to enroll for 15 credits and then blaming the faculty for their poor performance is outrageous.

Jay Coleman recently responded to a very similar question and therefore I am forwarding Jay’s response as it appropriately addresses this question:

I have a working version of a DFW model covering the entire campus (undergraduates), which includes a myriad of factors. Once I account for pre-entry characteristics, student demographics, course characteristics, the level of student engagement, instructor of the course, how much students are working off campus, etc., students attempting fewer than 15 hours actually do statistically significantly worse in their courses.  Every hour attempted below 15 is associated with about a 1.5% reduction in the odds of passing a class with an A, B, or C.

I can add that I’ve done a lot of other modelling of student success over the last 3-4 years, using a variety of outcome measures in a variety of areas, and taking less than 15 hours is frequently a negative factor. At worst, it’s a statistically insignificant one.

The question then obviously is why. Yes, there could be omitted factors involved that are skewing results, and we continue to pull in more data on more factors to help address such problems.  However, in virtually all of these models I’m accounting for the student’s academic preparation pre-entry (e.g., SAT/ACT scores and high school GPAs), which major they’re in, how much they’re working off campus, how much they’re engaged on campus (e.g., visits to the Library, Wellness Center, Game Room, Nature Trails) – i.e., things that would be expected to capture reasons why students might want to take fewer hours.

I think part of the answer is that we presume that students take the extra time that they would have been using on the fifth course and allocate it to their other four courses. Too often, they simply don’t.  Instead, that time is being spent on leisure (or other) activities instead, and depending on what those activities are, that in itself could be a bad thing.  I think another part of the answer is that hours attempted is a partial proxy for a construct that is increasingly showing up in educational research: grit.  Students with more grit – i.e., persistence, commitment, work ethic, etc. – simply do better than students with less, and high levels of grit make up for a lot.  If you’re trying fewer hours (all other things being equal), it’s just not a good sign regarding your level of grit.

Research continues, but signs are that the push for 15 has really helped the grad rate. Keep in mind that when we push taking 15, it’s for all FTICs at all points in their programs of study, not just the ones who just started.  Thus, it can have a positive and more immediate grad rate benefit for those who are late(r) in their programs.  Our 4-year grad rate has increased by nearly 50% over the last four years, from about 26% to 38.5%, and we expect next year’s number to be significantly higher.  Pushing 15 per term (or 30 per year) has a lot to do with that.

ISQ emphasis

Questioner: Anonymous

Question Posed to Pamela Chally, Interim Provost and VP of Academic Affairs

The recent emphasis on ISQ bothers me as an educator. Is it my role to educate the students or make them happy?  Please do not respond saying these are the same. If you want references and recent research, ISQ does not determine that. Many universities are on the path to make the ISQ result confidential and available to only the faculty while UNF posts them online by faculty and course number. Questions.

1. To what extent have you made sure that the improvement of ISQ is not coming at the cost of course quality? Many faculty are putting excessive weight on attendance, take home exams, open book exams, and grade curves so that students like them [the faculty]. How are colleges controlling that? Or is it a college’s policy to do only what is necessary to improve the metrics

and get state money? If so, can we change the mission statement to low quality instead of high quality education?

2. It has been proven that ISQ scores are discriminating against immigrants (especially with an accent) and women. Is it in UNF’s new mission to use a statistic that has been proven to be discriminating? If the administration thinks that is not true in the case of UNF, have they done a study to make certain?

3. If the administration is so concerned for students, why don’t we drop the full time student requirement from 15 to 12 credits? Most students at UNF are employed, often even more than 40 hours. So forcing them to enroll for 15 credits and then blaming the faculty for their poor performance is outrageous.

 

UNANSWERED

ISQ results and Promotion of Faculty

Questioner : Anonymous

Question posed to Pamela Chally, Interim Provost and VP of Academic Affairs and Kally Malcom – Bjorklund, UFF President

This question is for the UFF union president and the UNF administration interim Provost. Can each of you weigh in on the appropriateness — contractually and scientifically — of a Promotion and  Tenure Committee or an administrator to focus narrowly on ISQ results (student evaluations) and impose a threshold score for a particular item when making judgements of teaching?

Question answered by Pamela Chally, Interim Provost and VP of Academic Affairs

Kally and I thought we would answer that together. I really appreciate that, because it comes directly form the contract.

The Collective Bargaining agreement addresses criteria for demonstrating teaching effectiveness, and the process of applying for tenure and promotion. Article 20 addresses tenure:
“Judgments of academic excellence are complex. They cannot easily be reduced to a quantitative formula, nor can the considerations that must be applied in each individual case be completely described in general terms or by numbers alone, separate from necessary qualitative assessments.”

 

Article 20.5 is the section titled Criteria for Tenure and Basis for Tenure Decision.

This section includes the paragraph I just mentioned, and section D states that the tenure decision shall take into account annual assignments and annual performance evaluations, among other things.

 

Performance Evaluations are addressed in Article 18, and on teaching effectiveness. This article offers the most specific language about how faculty can demonstrate teaching effectiveness. And it is in Article 18 where ISQs are addressed specifically.

 

Article 18.2(d) University Required Student Evaluations.

“(1) The University required student Instructional Satisfaction Questionnaire (or ISQ) is one tool for evaluating teaching performance, and all the required ISQs must be included in the annual evaluation portfolio. However, the evaluation of a faculty member shall not be based solely or primarily on student evaluations if the faculty member has provided other information or evidence in support of his/her teaching performance.”

 

Article 18.4(a)1

“There are many approaches to and dimensions of pedagogical work. Thus, the evaluation of teaching performance shall consider the range of pedagogical activities engaged in by the faculty member.”

 

Another section in Article 18 offers 15 examples of pedagogical activities that can be used to evaluate teaching effectiveness, and ISQs are but one of the measures listed.

 

The Collective Bargaining agreement has language in Articles 18 and 20, as well as several promotion articles that outline the appropriate way to evaluate teaching effectiveness. The person asking the question also seeks scientific data supporting or challenging the use of student evaluations as the singular method of assessing teaching effectiveness. In the interest of brevity, I will not use this time to point to the several studies that are out there related to the usefulness of student evaluations. The best practices for evaluating faculty teaching are already addressed clearly within the collective bargaining agreement. And that is an agreement that is negotiated not just by UFF but also by the administration, and I was so happy that Provost Chally told us this morning that they are committed to following the contract, which we all must do, so we are happy to stand together in support of Articles 18, 20, and so many more.

 

Dr. Chally: It’s a multifaceted decision, and it’s really important that the individual make their case as to why they should be supported

ISQs and Promotion

Questioner: Anonymous

Question posed to David Fenner, FA President

Word is out there that UNF has denied tenure to faculty members who have only a little less than perfect scores on the ISQs even though the university committee’s evaluations were positive as were those that came earlier in the process. Can the FA Executive Committee bring a resolution to notify the BOT that this administration is destroying people’s lives at the university?

 

Answer from the floor by David Fenner, FA President

I will pass this on to Dr. Rakita, FA vice president and chair of Executive Committee, to pick up this question at the Executive Committee. Because this is a matter of terms of employment, the Union needs to take center stage on this rather than the Faculty Association. But having said that, we’ll certainly be involved with that. I’d also like to say that in the past, Faculty Association presidents have been strong voices of advocacy with regard to both faculty salaries and with promotion and tenure, where they sit as members of the BOT. I believe that all of that is going to come up at the June BOT meeting, and I will be there represented as appropriate.

Improving Course/Faculty Evaluations

Questioner: Anonymous

Posed to: Radha Pyati, President Faculty Association

Regarding course/faculty evaluations – Is there a group working on improving the questions so they move beyond a happiness index for students?

Response by Dan Richard, Office of Faculty Enhancement

The Faculty Enhancement Committee is in discussion about the administration and use of Instructional Satisfaction Questionnaires. Proposals for changes are not yet formalized.

ISQ method

Questioner: Matthew Corrigan

Posted to: Jay Coleman, Interim Associate Provost or Marianne Jaffee, Executive Assistant to the Provost

“I think it would be helpful, since we’re doing a new ISQ method, which we see if there are any differences between going totally online and what we did before, in terms of response rates, in terms of means on particular questions, in terms of demographics of students who are answering the questions. So, when you get a chance, could you try to make those comparisons, maybe from last fall – you know, the previous fall? That would be helpful. Thanks.”

In responding of Online ISQs from Dan Richard, Director Office of the Faculty Enhancement:

Response: From Dr. Dan Richard

In the Spring of 2014, the Provost asked Dan Richard, OFE Director, to establish a task force to address questions about the online ISQ method. The Task Force was organized to include representation from across colleges and affected units. The ISQ Task Force met twice in the Spring of 2014 and once (so far) in the Fall of 2014. Of major concern was the drop in response rates between face-to-face administration of the ISQ and the online version and the potential for this drop in response rates to affect the validity and subsequent interpretation of the instrument. Response rates for the face-to-face administration of the ISQ tended to range around 70%. The Online ISQ in the Fall of 2013 and Spring 2014 had response rates in the 50-60% range. At the end of the Spring 2014 semester, the Task Force indicated that more evidence was needed to determine the impact of the online administration and to address the impact of lower response rates. Although some analysis has been completed, additional analyses are being conducted currently. The Task Force also will address the ease of access for students to the Online ISQ and the length and content of the survey. The ISQ Task Force has meetings scheduled once every two weeks, including an upcoming meeting on October 8th at 1:30 p.m. in the OFE/FA Conference Room. Questions, comments, and suggestions for the Task Force should be forwarded to Dan Richard at drichard@unf.edu. A complete answer to Dr. Corrigan’s questions will be forwarded in about one month as a more complete analysis of the data is in its final stages.

ISQ Evaluations

Questioner: Anonymous

Posted to: Mark Workman, Provost Academic Affairs

“I would like to submit the following questions anonymously to the Faculty Association and ask for a written response from the administration.  It seems UNF posted the Student ISQ evaluations of faculty to comply with state laws.
1. Why were faculty not informed?
2. Why was only one item selected?
3. Why were the grade distributions posted as well? What is the rationale? Is it required by law to post those?
4. Why are faculty IDEA ratings of administrators not shared with faculty? Would those ratings not fall under the Florida sunshine law?  Even if it is not required by law, what is the reason not to make these ratings of administrators available, especially in light of greater transparency and accountability?”

 

Written response from Former Provost Mark Workman, Academic Affairs:

Gordon,

Please share this statement or not, verbally or in writing, as you see fit.

 

Thanks.

Mark

 

It is important that we remind or inform concerned faculty that the GDRs by instructor are grade distribution reports already posted for the university on the www.myedu.com website.  It is Academic Affairs’ understanding that for several years, students requested UNF’s participation in the myedu.com site – the Registrar would create a file and the General Counsel’s office would then send (or clear the Registrar to send) an  instructor-level grade distribution report to  myedu.com; this data was considered public information. Because instructor-level grade distribution reports were deemed to be public information, the Registrar was essentially responding to the student’s public records request by providing myedu.com with the data to upload on its site.  Once on the myedu.com site, students created (free) accounts to access this information.  Before Student Government approached Academic Affairs with its request, the SG President and Vice President researched other institutional websites and provided Academic Affairs  with examples of GDRs (Texas A&M and Indiana University-Bloomington) which made this data available to students on the home university’s website.  This would obviate the need for UNF to send this data to myedu.com and require students to create accounts.  SG requested the same ease of access to this data for UNF students.  Once Academic Affairs confirmed that the Registrar and General Counsel were providing this data to myedu.com and there were no issues of GDRs having limited access protection, we asked the SG President and Vice President to discuss their request with the President of Faculty Association.  Following that discussion, ITS and Enrollment Services worked with SG to post the data in a highly accessible and visible place on UNF’s website. The project was completed ahead of schedule thus the notification to students and faculty followed the posting of the data on UNF’s site.

 

With regard to the question of using aggregated data, it seemed impractical to offer to students this option since they were already accessing instructor-level data through myunf.edu.  To restrict the data they already had access to via myedu.com would not be responding to the request for this (public) data.   Academic Affairs suggests that representatives of the Faculty Association or General Counsel meet with Student Government to discuss the possibility of providing GDRs that are aggregated at the department level.  Since SG requested this data, it seems they would be the appropriate body to discuss other options.